Unicode to ASCII conversion issues (issues within Needle)

[tghosth]

I have deleted this comment and opened a new issue with a better description here as this issue is now focused on a different problem, see here.

[marco-lancini]

Yes, I was experiencing some issues as well. Can you please confirm you are using the latest version pulled from develop?

[tghosth]

ok so interestingly I just pulled the latest develop code from the mwrlabs repository and I got an error somewhere else, after selecting the discount bank application in binary/metadata. I think it is still related though.

root@kali:~/Work# python ./needle/needle/needle.py
             __   _ _______ _______ ______         _______
             | \  | |______ |______ |     \ |      |______
             |  \_| |______ |______ |_____/ |_____ |______
        
                   Needle v0.0.4 [mwr.to/needle]                  
  [MWR InfoSecurity (@MWRLabs) - Marco Lancini (@LanciniMarco)]   

[needle] > set IP 10.0.0.7
IP => 10.0.0.7
[needle] > set PORT 22
PORT => 22
[needle] > set VERBOSE True
VERBOSE => True
[needle] > set DEBUG True
DEBUG => True
[needle] > use binary/metadata
[needle][metadata] > run
[D] Setup local output folder: /root/.needle/output
[?] Attention! The folder chosen to store local output is not empty: /root/.needle/output
[?] Do you want to back it up first?
[?] Y: the content will be archived in a different location, then the folder will be emptied
[?] N: no action will be taken (destination files might be overwritten in case of filename clash)
[y/n]: 
[*] Checking connection with device...
[V] Connection not present, creating a new instance
[V] Setting up SSH connection...
[+] Connected to: 10.0.0.7
[V] Creating temp folder: /var/root/needle/
[D] [REMOTE CMD] Remote Command: if [ -d /var/root/needle/ ]; then echo "yes"; else echo "no" ; fi
[D] [REMOTE CMD] Remote Command: mkdir /var/root/needle/
[*] Target app not selected. Launching wizard...
[D] [REMOTE CMD] Remote Command: if [ -f /var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist ]; then echo "yes"; else echo "no" ; fi
[V] Refreshing list of installed apps...
[D] [REMOTE CMD] Remote Command: /bin/su mobile -c /usr/bin/uicache
[D] Copy the plist to temp: /var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist -> /var/root/needle/LastLaunchServicesMap.plist
[D] [REMOTE CMD] Remote Command: cp /var/mobile/Library/MobileInstallation/LastLaunchServicesMap.plist /var/root/needle/LastLaunchServicesMap.plist
[D] Converting plist to XML: /var/root/needle/LastLaunchServicesMap.plist
[D] [REMOTE CMD] Remote Command: plutil -convert xml1 /var/root/needle/LastLaunchServicesMap.plist
[D] Extracting content from: /var/root/needle/LastLaunchServicesMap.plist
[D] Sanitizing content from: /var/root/needle/LastLaunchServicesMap.plist
[D] Downloading: "/var/root/needle/LastLaunchServicesMap.plist" -> /root/.needle/tmp/plist
[D] [LOCAL CMD] Local Command: sshpass -p "alpine" scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 22 [email protected]:"/var/root/needle/LastLaunchServicesMap.plist" /root/.needle/tmp/plist
[D] Uploading: "/root/.needle/tmp/plist" -> "/var/root/needle/LastLaunchServicesMap.plist"
[D] [LOCAL CMD] Local Command: sshpass -p "alpine" scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 22 -r "/root/.needle/tmp/plist" [email protected]:"/var/root/needle/LastLaunchServicesMap.plist"
[D] [REMOTE CMD] Remote Command: cat /var/root/needle/LastLaunchServicesMap.plist
[D] Parsing plist content
[+] Apps found:
		0 - com.wanmei.mini.condorpp-532-8
		1 - il.co.discountbank.discountbankSME
		2 - com.ynet-internet.Ynet-iPad
Please select a number: 1
[+] Target app: il.co.discountbank.discountbankSME
[*] Retrieving app's metadata...
------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/hgfs/Work/needle/needle/core/framework/module.py", line 112, in do_run
    pre = self.module_pre()
  File "/mnt/hgfs/Work/needle/needle/core/framework/module.py", line 141, in module_pre
    if self.app_check() is None: return None
  File "/mnt/hgfs/Work/needle/needle/core/framework/framework.py", line 692, in app_check
    self.APP_METADATA = Framework.APP_METADATA = self.device.app.get_metadata(app)
  File "/mnt/hgfs/Work/needle/needle/core/device/app.py", line 20, in get_metadata
    return self._retrieve_metadata()
  File "/mnt/hgfs/Work/needle/needle/core/device/app.py", line 39, in _retrieve_metadata
    plist_local = self._device.remote_op.parse_plist(path_local)
  File "/mnt/hgfs/Work/needle/needle/core/device/remote_operations.py", line 227, in parse_plist
    self._device.printer.debug('Copy the plist to temp: {} -> {}'.format(plist, plist_copy))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 88-97: ordinal not in range(128)
------------------------------------------------------------
[!] UnicodeEncodeError: 'ascii' codec can't encode characters in position 88-97: ordinal not in range(128).
[needle][metadata] > 

I have also replicated the issue on another device (iPhone 6) so it doesn't seem to be a device specific problem. The error is basically identical.

[marco-lancini]

ok, I reverted the commits related to "[FIX] Invalid characters when parsing plist files".
Can you try pull develop now and see if you can at least get the app's metadata?
If yes, I'll try then to re-add the sanitize_plist function

[marco-lancini]

Reverting the changes fixed the issue.
I'll have another go to try and sanitize plist files.

[tghosth]

Hi @marco-lancini, please can this issue be reopened as it occurs again following the conversion to bplist.

[marco-lancini]

plist parsing errors might be because of this:
libimobiledevice/libplist#83

[tghosth]

Hi @marco-lancini, I have dug into this a little more and discovered the following. Unfortunately, it seems I have misdiagnosed this issue due to my inexperience with python.

The binary name in this app has Hebrew characters in it. Since we are using python 2, all string literals are assumed to be ascii. This means, if we try and do something like 'Binary name is: {}'.format(binary_name), it fails with an ascii error like UnicodeEncodeError: 'ascii' codec can't encode characters in position....

The ways I have seen to fix this are to append a "u" to every string literal in use or to add from __future__ import unicode_literals at the top of every file. The latter especially could have unanticipated side effects and neither of these methods seems very practical. Do you have any other insights?

[marco-lancini]

@tghosth, you might want to have a look at this as well: #106 (comment)

[tghosth]

Hi @marco-lancini, I had a quick look at #106 (comment) but that seems to be a separate problem.

Do you have any thoughts on how we can fix this unicode issue? What if we added from __future__ import unicode_literals to the top of every file but inside a conditional that allows disabling the import if it causes other issues further down the line?

[marco-lancini]

Ok I finally had the time to get a proper look: you're right that's a different issue.

Regarding the unicode issue, as you said adding from __future__ import unicode_literals to the top of every file could have a pretty massive impact and unforeseen side effects...

What about treating (forcing) just the content of plist files as unicode?

[tghosth]

I cannot remember if there were other places where I was having unicode issues but plists would be a good place to start, I will try and look adding this when I get some time :)

[tghosth]

Just an update that I have looked at this a little more and it is really a pervasive problem wherever there are non-ascii characters. Unfortunately, just treating plists as unicode won't help.

I have prepared a branch which adds __future__ import unicode_literals to the top of every file.

If you can do testing with it, that would be great. This is an example of an app which won't work without the unicode support.

In the meantime, I also want to do some more testing to make sure it doesn't break other stuff before I open a PR.

[tghosth]

It appears there are two issues regarding Unicode. One relates to processing within needle as shown here in this example

I think this type of issue needs to be fixed by adding __future__ import unicode_literals to the top of every file which I am working on as noted here.

The second issue is within paramiko which seems to be the cause of the initial issue and I have opened a new issue to discuss this #118.

[whoot]

Still having this issue with the current release (Needle v1.3.2 [mwr.to/needle])!

[needle][list_apps] > use storage/data/files_plist
[+] Resource file successfully loaded
[needle][files_plist] > run
[*] Checking connection with device...
[+] Already connected to: 192.168.1.137
[+] Target app: com.redacted.for.privacy
[*] Retrieving app's metadata...
[!] UnicodeEncodeError: 'ascii' codec can't encode characters in position 83-84: ordinal not in range(128)