网站经营与建设,手机开发者选项怎么设置手机流畅,成都科技网站建设联系电话,如何开发电商网站最近在做一个基于无障碍自动刷短视频的APP#xff0c;需要支持用任意蓝牙遥控器远程控制#xff0c; 把无障碍服务流程大致研究了一下#xff0c;从下面3个部分做一下小结。
1、需要可调整自动上滑距离和速度以适配不同的屏幕和应用
智能适配99%机型#xff0c;滑动参数可…最近在做一个基于无障碍自动刷短视频的APP需要支持用任意蓝牙遥控器远程控制 把无障碍服务流程大致研究了一下从下面3个部分做一下小结。
1、需要可调整自动上滑距离和速度以适配不同的屏幕和应用
智能适配99%机型滑动参数可自由调节。
默认的距离和速度可能在个别手机上无法达到滑屏的要求表现就是屏幕可见滑动了下但还是停留回当前界面。所以需要给用户一种自定义调整的方式这里以【辅助触控】APP为例提供了屏幕配置的实现可以自行拖动滑动起始点然后调整滑动参数。 编辑屏幕的时候支持增删按键映射12个可编程功能键 (F1-F12)并自定义如下参数
➕ 单击/双击/连击/长按⚙️ 自定义间隔0.1-30秒⏱️ 按压停留时长设置
支持手势轨迹定制如下参数 起点/终点坐标设置⏲️ 自定义间隔3-30秒和滑动速度 四向滑动独立配置
在定义好按键映射后还可以对其进行组合控制编写一组相关动作然后执行。 其中还可以进一步定义文本识别后要执行的动作比如单击文本节点、返回、上滑等。 2、监听TYPE_WINDOW_STATE_CHANGED事件在图形验证码出现时停止需要能识别出带有关键文本的视图元素
比如支付宝看视频领红包活动会一定机率跳出图形验证码需要用户手动点选如果此界面继续滑屏很容易被系统识别到正在进行自动化脚本刷屏。需要对界面内容识别比如文本 “请依次点击下面的图案”。 在自动滑屏期间检测到该事件说明有窗口焦点切换一般就是切换到了不同的窗口比如 Dialog、PopupWindow等。有可能就是这个验证框这时候我们需要拿到getRootInActiveWindow()然后通过无障碍API findAccessibilityNodeInfosByText找出包含上面文本的Node。 //这里是我们要找的可能的文本
val ocrTexts listOf(请在下图依次点击)
for(ocrTry in 0 until 4) {for (text in ocrTexts) {//找包含text的那些节点这些节点要么是能呈现指定文本(text、hint)的视图要么是包含指定内容描述content description的视图var nodes rootInActiveWindow?.findAccessibilityNodeInfosByText(text)nodes?.forEach { nodeInfo -//找到了验证框停止滑屏并发出声音和震动提示用户需要手动验证。autoRepeatIntervalJob?.cancel()playBeepSoundAndVibrate(5000)return}}if (ocrTry 3) {//延迟一下有可能文本内容还没加载Thread.sleep(500)}
}在循环中每次我们都重新获取rootInActiveWindow 否则可能获取到的不是当前界面不用担心性能问题只要没有新的TYPE_WINDOW_STATE_CHANGED事件发生都会使用缓存。所以每次获取的好处就是即使事件发生了我下一个循环就能得到新界面。
实测发现如下图支付宝这个验证框使用findAccessibilityNodeInfosByText居然找不到 难道他是图片带着怀疑我用uiautomatorviewer看了下布局发现只是一个TextView, 文本也是“请在下图依次点击”和我们检索字符串一样只是它的ImportantForAccessibility属性是false 我们的AccessibilityService的config里也添加了flagIncludeNotImportantViews包括不重要的视图照理应该能找到并返回。然后我又尝试了下遍历的方式
fun AccessibilityService.findTextByTraversal(text: String, include: Boolean false): ListAccessibilityNodeInfo {val result mutableListOfAccessibilityNodeInfo()traverseNodes(result, rootInActiveWindow, text, include)return result
}
private fun traverseNodes(result: MutableListAccessibilityNodeInfo,node: AccessibilityNodeInfo?,searchText: String,include: Boolean false,
) {node?.let {if (node.text ! null node.text.isNotEmpty()) {if (include node.text.contains(searchText)) {result.add(node)} else if (node.text searchText) {result.add(node)}if (DebugUtils.DEBUG) DebugUtils.logD(TAG, traverseNodes find $node)}for (i in 0 until node.childCount) {traverseNodes(result, node.getChild(i), searchText, include)}}
}
竟然能找到这个node这样的话先修改一下逻辑优先使用findAccessibilityNodeInfosByText找不到再递归找。
第3部分我们通过源码梳理一下AccessibilityService和AccessibilityManagerService之间的通信过程尝试分析一下findAccessibilityNodeInfosByText是怎样进行查找的 3、AccessibilityService和AccessibilityManagerService之间的通信过程
参考源码
https://xrefandroid.com/android-11.0.0_r48/
首先我们需要简单了解一下AccessibilityService启动流程。
AccessibilityService启动流程
在SystemServer主进程服务启动阶段AccessibilityManagerServiceAMS作为系统服务被初始化负责管理全局无障碍服务生命周期及事件分发。
// frameworks/base/services/java/com/android/server/SystemServer.java
private static final String ACCESSIBILITY_MANAGER_SERVICE_CLASS com.android.server.accessibility.AccessibilityManagerService$Lifecycle;
private void startOtherServices(NonNull TimingsTraceAndSlog t) {...try {mSystemServiceManager.startService(ACCESSIBILITY_MANAGER_SERVICE_CLASS);} catch (Throwable e) {reportWtf(starting Accessibility Manager, e);}}
实例化AccessibilityManagerService$Lifecycle对象并调用其onStart() 将AMS发布出来之后就可以通过Context.getSystemService(Context.ACCESSIBILITY_SERVICE)获取对应的AccessibilityManager来和AMS通信。
AMS init初始化时注册 PackageMonitor 监听应用安装/卸载事件动态维护已注册的无障碍服务列表注册ACTION_USER_PRESENT读取所有已安装应用的无障碍服务信息查询所有已安装应用中的无障碍服务信息
//frameworks/base/services/accessibility/java/com/android/server/accessibility/AccessibilityManagerService.java 其中只要有somethingChanged true, 比如下面这个读取到已安装应用发生了变更 则调用onUserStateChangedLocked更新相关信息 包括绑定App的AccessibilityService, 创建一个AccessibilityServiceConnection来绑定服务和完成两者之间的通信。 绑定成功后会回调onServiceConnected(ComponentName componentName, IBinder service) 我们知道Service的绑定过程是被绑定的服务会启动然后在onBind(Intent)返回一个IBinder对象给绑定者, 绑定者在onServiceConnected中可以获取到一个用于和Service进行IPC通信的接口对象IBinder。
比如前面提到的TYPE_WINDOW_STATE_CHANGED事件传递过程为: AMS sendAccessibilityEvent(AccessibilityEvent event, int userId) - AMS notifyAccessibilityServicesDelayedLocked(AccessibilityEvent event, boolean isDefault) - AccessibilityServiceConnection.notifyAccessibilityEvent(event) - mServiceInterface.onAccessibilityEvent(event, serviceWantsEvent) ···IPC··· IAccessibilityServiceClientWrapper.onAccessibilityEvent(event, serviceWantsEvent) - AccessibilityService.onAccessibilityEvent(event) 上面是AMS到AccessibilityService的通信AccessibilityService到AMS则是通过AccessibilityInteractionClient。
前面已经提到在onBind回调的时候我们返回了一个IAccessibilityServiceClientWrapper IBinder给AMS AMS在绑定服务成功后拿到service IBinder调用了initializeService将AMS端的AccessibilityServiceConnection回传给了AccessibilityService如下代码所示
public void onServiceConnected(ComponentName componentName, IBinder service) {...mServiceInterface IAccessibilityServiceClient.Stub.asInterface(service); //service就是IAccessibilityServiceClientWrapper IBinder...mMainHandler.sendMessage(obtainMessage(AccessibilityServiceConnection::initializeService, this));...
}
private void initializeService() {...serviceInterface.init(this, mId, mOverlayWindowTokens.get(Display.DEFAULT_DISPLAY));...
}//frameworks/base/core/java/android/accessibilityservice/AccessibilityService$IAccessibilityServiceClientWrapper
public void init(IAccessibilityServiceConnection connection, int connectionId, IBinder windowToken) {Message message mCaller.obtainMessageIOO(DO_INIT, connectionId, connection, windowToken);mCaller.sendMessage(message);
}public void executeMessage(Message message) {...case DO_INIT: {mConnectionId message.arg1;SomeArgs args (SomeArgs) message.obj;IAccessibilityServiceConnection connection (IAccessibilityServiceConnection) args.arg1;IBinder windowToken (IBinder) args.arg2;args.recycle();if (connection ! null) {//关联 IAccessibilityServiceConnectionAccessibilityInteractionClient.getInstance().addConnection(mConnectionId, connection);mCallback.init(mConnectionId, windowToken);mCallback.onServiceConnected();}...}...
} IAccessibilityServiceClientWrapper 将AMS传来的IAccessibilityServiceConnection添加到AccessibilityInteractionClient中缓存起来后续用来和AMS通信。 到这里我们的AccessibilityService与AMS的通道就建好了 AccessibilityService - AccessibilityInteractionClient - IAccessibilityServiceConnection ···IPC··· AccessibilityServiceConnection - AMS 现在回头来看findAccessibilityNodeInfosByText 一般我们需要先getRootInActiveWindow获取root节点。 getRootInActiveWindow获取root节点
public AccessibilityNodeInfo getRootInActiveWindow() {return AccessibilityInteractionClient.getInstance().getRootInActiveWindow(mConnectionId);}//AccessibilityInteractionClient
public AccessibilityNodeInfo getRootInActiveWindow(int connectionId) {//这里使用固定的ACTIVE_WINDOW_ID和ROOT_NODE_ID在AMS那边会对应到当前可交互窗口的rootreturn findAccessibilityNodeInfoByAccessibilityId(connectionId,AccessibilityWindowInfo.ACTIVE_WINDOW_ID, AccessibilityNodeInfo.ROOT_NODE_ID, false, AccessibilityNodeInfo.FLAG_PREFETCH_DESCENDANTS, null);
}public Nullable AccessibilityNodeInfo findAccessibilityNodeInfoByAccessibilityId(int connectionId, NonNull IBinder leashToken, long accessibilityNodeId,boolean bypassCache, int prefetchFlags, Bundle arguments) {if (leashToken null) {return null;}int windowId -1;try {//获取前面关联的缓存在线程中的IAccessibilityServiceConnectionIAccessibilityServiceConnection connection getConnection(connectionId);if (connection ! null) {windowId connection.getWindowIdForLeashToken(leashToken);} else {if (DEBUG) {Log.w(LOG_TAG, No connection for connection id: connectionId);}}} catch (RemoteException re) {Log.e(LOG_TAG, Error while calling remote getWindowIdForLeashToken, re);}if (windowId -1) {return null;}return findAccessibilityNodeInfoByAccessibilityId(connectionId, windowId,accessibilityNodeId, bypassCache, prefetchFlags, arguments);
}public AccessibilityNodeInfo findAccessibilityNodeInfoByAccessibilityId(int connectionId,int accessibilityWindowId, long accessibilityNodeId, boolean bypassCache,int prefetchFlags, Bundle arguments) {...//向AMS connection发请求 传入AccessibilityInteractionClient自身作为callback用于接收结果回调packageNames connection.findAccessibilityNodeInfoByAccessibilityId(accessibilityWindowId, accessibilityNodeId, interactionId, this,prefetchFlags, Thread.currentThread().getId(), arguments);...//等待AMS 返回结果ListAccessibilityNodeInfo infos getFindAccessibilityNodeInfosResultAndClear(interactionId);...
}//frameworks/base/services/accessibility/java/com/android/server/accessibility/AbstractAccessibilityServiceConnection.java
public String[] findAccessibilityNodeInfoByAccessibilityId(int accessibilityWindowId, long accessibilityNodeId, int interactionId,IAccessibilityInteractionConnectionCallback callback, int flags,long interrogatingTid, Bundle arguments) throws RemoteException {...// 解析当前的windowIdresolvedWindowId resolveAccessibilityWindowIdLocked(accessibilityWindowId);...//找到当前窗口和AMS之间的交互连接对象connection mA11yWindowManager.getConnectionLocked(mSystemSupport.getCurrentUserIdLocked(), resolvedWindowId);...//将请求通过IPC发给连接的远程端connection.getRemote().findAccessibilityNodeInfoByAccessibilityId(accessibilityNodeId, partialInteractiveRegion, interactionId, callback,mFetchFlags | flags, interrogatingPid, interrogatingTid, spec, arguments);...
}
这里经过查阅源码发现connection是应用通过ViewRootImpl创建新窗口如 Activity、Dialog、PopupWindow 等时会通过IPC向AMS进行addAccessibilityInteractionConnection() 调用从而注册窗口与AMS之间的交互连接对象
connection.getRemote()就是这个对象 如下源码所示的AccessibilityInteractionConnection
//frameworks/base/core/java/android/view/ViewRootImpl.java
public void setView(View view, WindowManager.LayoutParams attrs, View panelParentView,int userId) {...if (mAccessibilityManager.isEnabled()) {mAccessibilityInteractionConnectionManager.ensureConnection();}...
}final class AccessibilityInteractionConnectionManagerimplements AccessibilityStateChangeListener {...public void ensureConnection() {final boolean registered mAttachInfo.mAccessibilityWindowId! AccessibilityWindowInfo.UNDEFINED_WINDOW_ID;if (!registered) {mAttachInfo.mAccessibilityWindowId mAccessibilityManager.addAccessibilityInteractionConnection(mWindow,mLeashToken,mContext.getPackageName(),new AccessibilityInteractionConnection(ViewRootImpl.this));}}...
}static final class AccessibilityInteractionConnection extends IAccessibilityInteractionConnection.Stub {private final WeakReferenceViewRootImpl mViewRootImpl;AccessibilityInteractionConnection(ViewRootImpl viewRootImpl) {mViewRootImpl new WeakReferenceViewRootImpl(viewRootImpl);}...Overridepublic void findAccessibilityNodeInfoByAccessibilityId(long accessibilityNodeId,Region interactiveRegion, int interactionId,IAccessibilityInteractionConnectionCallback callback, int flags,int interrogatingPid, long interrogatingTid, MagnificationSpec spec, Bundle args) {ViewRootImpl viewRootImpl mViewRootImpl.get();if (viewRootImpl ! null viewRootImpl.mView ! null) {viewRootImpl.getAccessibilityInteractionController().findAccessibilityNodeInfoByAccessibilityIdClientThread(accessibilityNodeId,interactiveRegion, interactionId, callback, flags, interrogatingPid,interrogatingTid, spec, args);} else {// We cannot make the call and notify the caller so it does not wait.try {callback.setFindAccessibilityNodeInfosResult(null, interactionId);} catch (RemoteException re) {/* best effort - ignore */}}}...}
所以最终AMS会调用到了应用端同时传递了回调callback用于接收结果
viewRootImpl.getAccessibilityInteractionController().findAccessibilityNodeInfoByAccessibilityIdClientThread(accessibilityNodeId,interactiveRegion,interactionId, callback, flags, interrogatingPid,interrogatingTid,spec,args);
//frameworks/base/core/java/android/view/AccessibilityInteractionController.java 我们之前在AccessibilityService中调用getRootInActiveWindow使用的accessibilityId是AccessibilityNodeInfo.ROOT_NODE_ID这里得到的就是mViewRootImpl.mView即窗口的根视图DecorView。 如果此时root view已经可见则封装并返回root的无障碍节点信息:
//frameworks/base/core/java/android/view/AccessibilityInteractionController$AccessibilityNodePrefetcher
public void prefetchAccessibilityNodeInfos(View view, int virtualViewId, int fetchFlags,ListAccessibilityNodeInfo outInfos, Bundle arguments) {...AccessibilityNodeInfo root view.createAccessibilityNodeInfo();if (root ! null) {...outInfos.add(root);...}...
}之后将找到的节点通过回调callback.setFindAccessibilityNodeInfosResult(infos, interactionId)传回给请求方。
AMS端传递的callback对应的是AccessibilityService端的AccessibilityInteractionClient这个Binder 结果也就传到了AccessibilityInteractionClient即IPC调用过程如下
发起请求
AccessibilityService - IAccessibilityServiceConnection ···IPC··· AccessibilityServiceConnection - AMS - IAccessibilityInteractionConnection ···IPC··· AccessibilityInteractionConnection - 当前窗口应用的ViewRootImpl
返回结果
当前窗口应用的ViewRootImpl - callback ···IPC··· AccessibilityService 返回的是一个列表我们使用第一个作为找到的root节点。 findAccessibilityNodeInfosByText
和前面的IPC调用过程一样我们直接去ViewRootImpl去找对应的方法
Override
public void findAccessibilityNodeInfosByText(long accessibilityNodeId, String text,Region interactiveRegion, int interactionId,IAccessibilityInteractionConnectionCallback callback, int flags,int interrogatingPid, long interrogatingTid, MagnificationSpec spec) {ViewRootImpl viewRootImpl mViewRootImpl.get();if (viewRootImpl ! null viewRootImpl.mView ! null) {viewRootImpl.getAccessibilityInteractionController().findAccessibilityNodeInfosByTextClientThread(accessibilityNodeId, text,interactiveRegion, interactionId, callback, flags, interrogatingPid,interrogatingTid, spec);} else {// We cannot make the call and notify the caller so it does not wait.try {callback.setFindAccessibilityNodeInfosResult(null, interactionId);} catch (RemoteException re) {/* best effort - ignore */}}
}//AccessibilityInteractionController.java
private void findAccessibilityNodeInfosByTextUiThread(Message message) {...ListAccessibilityNodeInfo infos null;final View root findViewByAccessibilityId(accessibilityViewId);ArrayListView foundViews mTempArrayList;foundViews.clear();//首先找出包含检索字符串的viewroot.findViewsWithText(foundViews, text, View.FIND_VIEWS_WITH_TEXT| View.FIND_VIEWS_WITH_CONTENT_DESCRIPTION| View.FIND_VIEWS_WITH_ACCESSIBILITY_NODE_PROVIDERS);if (!foundViews.isEmpty()) {infos mTempAccessibilityNodeInfoList;infos.clear();final int viewCount foundViews.size();for (int i 0; i viewCount; i) {//依次遍历找到的view满足条件则生成AccessibilityNodeInfo添加到结果列表中View foundView foundViews.get(i);if (isShown(foundView)) {provider foundView.getAccessibilityNodeProvider();if (provider ! null) {//这里最可能是隐藏不重要节点的地方通过自定义AccessibilityNodeProvider实现返回null或者空即可ListAccessibilityNodeInfo infosFromProvider provider.findAccessibilityNodeInfosByText(text,
AccessibilityNodeProvider.HOST_VIEW_ID);if (infosFromProvider ! null) {infos.addAll(infosFromProvider);}} else {infos.add(foundView.createAccessibilityNodeInfo());}}}}...//通知callback结果updateInfosForViewportAndReturnFindNodeResult(infos, callback, interactionId, spec, interactiveRegion);}
主要看root.findViewsWithText
//ViewGroup实现 //View默认实现 默认情况下View 类的 getAccessibilityNodeProvider() 返回 null。 //TextView实现
根据代码或者注释我们知道了匹配规则系统会遍历View树只要view可见定义了content description或text并且包含我们要查找的文本忽略大小写这个view就认为是需要的。所有符合条件的view依次封装为AccessibilityNodeInfo添加到结果列表infos中: infos.add(foundView.createAccessibilityNodeInfo()); 之后返回结果给callback。 callback.setFindAccessibilityNodeInfosResult(infos, interactionId); 到目前为止并没有看到根据view的importantForAccessibilityno来过滤视图唯一可能得地方就是foundView自定义了AccessibilityNodeProvider进行了过滤如源码所示
provider foundView.getAccessibilityNodeProvider();
if (provider ! null) {ListAccessibilityNodeInfo infosFromProvider provider.findAccessibilityNodeInfosByText(text, AccessibilityNodeProvider.HOST_VIEW_ID);if (infosFromProvider ! null) {infos.addAll(infosFromProvider);}
}
只要provider.findAccessibilityNodeInfosByText此时返回null即可。
而遍历节点树的方式只要我们的AccessibilityServie申明了包含不重要视图这个flag View就能在节点树里找到。
