https://lwn.net/SubscriberLink/979671/4fb7c1827536d1ae/ LWN.net Logo LWN .net News from the source LWN * Content + Weekly Edition + Archives + Search + Kernel + Security + Events calendar + Unread comments + ------------------------------------------------------------- + LWN FAQ + Write for us User: [ ] Password: [ ] [Log in] | [Subscribe] | [Register] Subscribe / Log in / New account Python grapples with Apple App Store rejections [LWN subscriber-only content] By Joe Brockmeier June 27, 2024 An upgrade from Python 3.11 to 3.12 has led to the rejection of some Python apps by Apple's app stores. That led to Eric Froemling submitting a bug report against CPython. That, in turn, led to an interesting discussion among Python developers about how far the project was willing to go to accommodate app store review processes. Developers reached a quick consensus, and a solution that may arrive as soon as Python 3.13. The problem at hand is that Apple's macOS App Store is automatically rejecting apps that contain the string "itms-services". That is the URL scheme for apps that want to ask Apple's iTunes Store to install another app. Software distributed via Apple's macOS store is sandboxed, and sandboxed apps are prohibited from using URLs with the itms-services scheme. That string is in the urllib parser in Python's standard library, though an application may never actually use the itms-services handler. Of course, Apple did not do anything so straightforward as to explain this to Froemling. Once he filed an appeal with Apple about the rejection, Apple finally told him that parse.py and parse.pyc were the offending files. After that, he said, it was not hard to track down the problem: Now in retrospect I'm frustrated I didn't think to run a full text search for itms-services over Python itself earlier or stumble across one of the other cases of folks hitting this. Russell Keith-Magee started the discussion in the Python Core Development discussion forum on June 17. He wanted to know whether " `acceptable to app stores'" should be a design goal for CPython, or if that compliance should be a problem left to the tools that generate application bundles for app stores. Paranoid and inscrutable Keith-Magee noted in his initial question that Apple's review processes were the most "`paranoid and inscrutable'" of app-store-review processes, but that other app stores also had " `validation and acceptance processes that are entirely opaque'". One solution might be to obfuscate the offending string to pass review, but that might "`lead to an obfuscation arms race'" and there were no guarantees this would be the last time the project had to resolve app-validation problems. The other option, he said, was to consider this to be a distribution problem and leave it to tools like Briefcase, py2app, and buildozer to solve. Traditionally, they have had to patch CPython anyway, he said, because it did not support Android or iOS "out of the box". But that will change with Python 3.13 when no patching should be required for those platforms. Like what you are reading? Try LWN for free for 1 month, no credit card required. Alex Gaynor suggested that the project try a an approach that Keith-Magee had not put forward inspired by Gaynor's experience with the cryptography library. The project often receives complaints that the library refuses to parse a certificate that is technically invalid, but was in wide use. He said that the policy was to accept pull requests that work around those issues "`provided they are small, localized, and generally aren't too awful'". But, he added, these patches should only be accepted on the condition that someone complains to the third party (in this case Apple), and extracts some kind of commitment that they would do something about it. He suggested that the workaround be time-limited, to give users a decent experience "`while also not letting large firms simply externalize their bizarre issues onto OSS projects'". Brandt Bucher wondered whether obfuscation was even allowed, or if it would be seen as circumventing the review process. That was a question no one seemed to have an answer to; and Keith-Magee responded with an 8-Ball emoji and the phrase "`ask again later'." He added that Gaynor's approach sounded appealing, but it would be like screaming into the void. Apple, he said, barely has an appeals process and there is no channel available to the Python project "`to raise a complaint that we could reasonably believe would result in a change of policy'". Another approach, suggested by Alyssa Coghlan, would be to use a JSON configuration file that urllib would read to set up its module level attributes "`rather than hardcoding its knowledge of all the relevant schemes'". That could allow app generators to drop "itms-services" from the configuration file rather than patching urllib.py directly. Keith-Magee said that could work, but "`it strikes me as a bit of overkill for an edge case'" that could be handled by obfuscation or distribution-level patching. On June 20, Keith-Magee wrote that he had thought of another approach: adding a build-time option called "--with-app-store-patch" that removes code that is known to be problematic. He said it would be enabled by default for the iOS platform, and disabled elsewhere. It could be used when building an application for macOS, if the developer intended to distribute that application via the macOS App Store. He suggested that the option could also accept a path to a file with a patch, to allow distributors to provide an updated patch if an app store changes its rules after the maintenance window for a Python release has closed. Let's paint the bikeshed Coghlan asked if it was now time to "`paint a config option bikeshed'". She said that the proposed option name was both too broad and too narrow. The "app-store" component of the name was too broad, because it could encompass any app store, not only Apple app stores. The "patch" component was too narrow, because patch specifies the method of complying with policies rather than intent. There may be other methods required to comply with app-store-compliance checks. Keith-Magee liked the suggestion about dropping "patch" from the option name, and suggested painting the bikeshed a nice shade of "--with-app-store-compliance" that would interact with platform identification to sort out what is required. On June 25, Keith-Magee thanked participants in the discussion for their input, and pointed to a pull request that would implement the --with-app-store-compliance configuration option. In the request, he noted that it would be possible to use the option with platforms other than iOS or macOS, but there were no use cases for that at present. If all goes well, it should be available in Python 3.13. It is frustrating that free-software projects like Python have to waste time finding ways around opaque review processes just so developers can write software for non-free platforms. However, the approach taken by Keith-Magee and other CPython developers seems to be the least-bad option that offers the best experience for Python application developers. It will almost certainly not be the last time that a project runs into this problem. [Send a free link] ----------------------------------------- (Log in to post comments) Alternatively Posted Jun 27, 2024 14:06 UTC (Thu) by dskoll (subscriber, #1630) [ Link] (5 responses) I understand the desire of free software advocates to have our software run everywhere, but at some point maybe we shouldn't continue enabling these kinds of shenanigans? Every piece of application software that runs on Apple's ecosystem makes it a bit more valuable. How about if we stop playing Apple's games and not support their platform until they become reasonable and transparent? Every software project that did that would make Apple's ecosystem a bit less valuable and eventually this would be noticed. [Reply to this comment] Alternatively Posted Jun 27, 2024 14:45 UTC (Thu) by shironeko (subscriber, # 159952) [Link] What ends up happens is that people start shipping slightly different workarounds in downstream. The amount of apple software would not decrease, it just gets more frustrating to use free software on it. [Reply to this comment] Alternatively Posted Jun 27, 2024 14:55 UTC (Thu) by khim (subscriber, #9252) [Link ] (2 responses) While pretty interesting, I think that's the question for another time. There are absolutely no need to have string itms-services in your app if you don't support said services. And both "we want to support Apple" and "we don't support Apple" stances lead to the same outcome: removal of these from the standard library. Now, if you want to both support non-developers who want to use these proprietary services and also developers who get rejection notices because of them then you are in bind, but I think simple and straightforward removal is the best outcome: remove the support, leave these for guys who actually want to use these services from Python to resolve, somehow. They decided to deal with unreasonable company, they can deal with the fallout. If would be time to decide when Apple would bad something because it supports some other free service. [Reply to this comment] Alternatively Posted Jun 27, 2024 15:45 UTC (Thu) by flussence (subscriber, #85566) [Link] (1 responses) This is a general purpose URL parser. Saying it should not recognise an URL pattern used by a malevolent actor is like saying tzdata should delete timezones only used in "bad" countries. Anyway there's a really simple workaround for this. Simply ship the source file in EBCDIC. The type of whiteboard-interview weenies who come up with and enforce these restrictions will not be smart enough to figure that one out. [Reply to this comment] Alternatively Posted Jun 27, 2024 16:28 UTC (Thu) by notriddle (subscriber, # 130608) [Link] > This is a general purpose URL parser. Other general-purpose URL parsers don't special-case this scheme. Both URL specs have support for schemes that are unknown to the parser, and neither https://url.spec.whatwg.org/#special-scheme nor https://datatracker.ietf.org/doc/html/rfc1738#section-3 includes itms-services in their lists. [Reply to this comment] Alternatively Posted Jun 27, 2024 16:40 UTC (Thu) by elw (subscriber, #86388) [Link ] > How about if we stop playing Apple's games and not support their platform until they become reasonable and transparent? The problem I see with that position is that Python is not the only easy to use language for app development. The only thing that would be accomplished by standing firm and refusing to play their game would be a reduction in usage in the platform. Unless there is some Killer App (tm)[?], written in Python, that would cause tangible damage to the overall Apple ecosystem, Apple holds all the cards and they know it. [Reply to this comment] Strip the functionality altogether. Posted Jun 27, 2024 14:45 UTC (Thu) by pavon (subscriber, #142617) [ Link] (2 responses) I'd question whether something like itms-services scheme support really belongs in a standard library to begin with. That is a pretty niche functionality for a single operating system. Most python software running on macOS will never use it, just things like installers. There isn't equivalent support for the dozens of Windows-specific schemes that Microsoft uses. Obviously, you'd need to give time to deprecate it, so another work-around would be needed for the short-term, but in the long-run if Apple wants to make it hard to integrate with their platform, then stop supporting those integrations and let application developers use a third-party library. [Reply to this comment] Costs vs Benefits Posted Jun 27, 2024 16:35 UTC (Thu) by nickodell (subscriber, # 125165) [Link] (1 responses) I think that when asking whether to remove it, we should ask what the cost of the feature is. If you look at the PR which adds this support, excluding tests, it is added in one line of code. (https:// github.com/python/cpython/pull/104312) This seems like pretty minimal maintenance burden. [Reply to this comment] Costs vs Benefits Posted Jun 27, 2024 20:26 UTC (Thu) by pavon (subscriber, #142617) [ Link] The cost was negligible, until Apple decided to make it more expensive by requiring different behavior and thus different builds depending on how the user installed the app. [Reply to this comment] Alex's approach Posted Jun 27, 2024 16:06 UTC (Thu) by tialaramex (subscriber, # 21167) [Link] (6 responses) The choice to time-limit workarounds allows you to properly frame the problem when it inevitably arises. For example TLS 1.3 as designed has Downgrade prevention, if you try to *pretend* by meddling with the TCP data that somebody else's server doesn't know TLS 1.3, then either you tip off the server that you're interfering or you tip off the client, or both. There is no way to get to a scenario where the client thought it asked "Do you speak TLS 1.3?" but the server thought the client asked only "Do you speak TLS 1.2?" without the session being torn down. But, of course idiot middleboxes broke this. Because they don't implement TLS properly, they'd hide the client's real question about TLS 1.3 from the server, but they'd pass through its answer to the question they did ask, thus exactly performing a downgrade, which sets off the downgrade prevention, your browser just closes the session. This was discovered after TLS 1.3 shipped, because Downgrade prevention obviously can't be enabled pre-standard, that's a DOS against yourself in the standards development process. So, Google shipped builds of Chrome with a hack, _if_ you turned on the hack and _if_ the server says "I'm a TLS 1.3 server but you claimed not to know TLS 1.3 so that's fine, unless you were downgraded" this Chrome ignores the problem. Temporarily. This damages your security, but you're using a middle box, you already don't have any security, that's what you paid money for, to destroy your security. Google told the affected vendors that their customers could enable this hack (in their installed Chrome builds) and *crucially* it told them that in a fixed period of time this hack would just be summarily deleted from Chrome. They were not going to reify this nonsense as part of the protocol design, it was a temporary hack to get some people working while these companies fixed their shit. And they did it. Years ago now Chrome just deleted the hack. If your middle box vendor didn't fix their bug in a reasonable timeframe, or you couldn't be bothered to install their fix - now Chrome doesn't work, too bad. Everybody gets reliable TLS 1.3 Without making it clear that this stops working in (say) 12 months, there is *no* incentive for Apple to fix it, for Apple's developers to care that it's a problem, for anybody to do anything except dump on the Python maintainers whatever they can't be bothered to fix. They're making their own lives worse. If they set a specific deadline and stick to it, it's clear what the problem is, that doesn't mean Apple will care, but too bad, you offload the problem onto exactly the people who wanted it, Apple developers, people who are used to and have accepted that they have an unhealthy masochistic relationship to an unfeeling corporation. There is no reason why anybody involved in Python who doesn't share that weird relationship should be inconvenienced. [Reply to this comment] Alex's approach Posted Jun 27, 2024 16:20 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (5 responses) > Without making it clear that this stops working in (say) 12 months, there is *no* incentive for Apple to fix it, for Apple's developers to care that it's a problem, for anybody to do anything except dump on the Python maintainers whatever they can't be bothered to fix. They're making their own lives worse. The question is, why would Apple have any incentive to fix it? In the middlebox case, the makers get money and have support contracts with whoever buys their products. In Apple case, app developers have no alternatives than Apple if they want their app to run on iPhone/iPad, and Apple gets money from users, not from app developers. So the odds are completely stacked against the app developers, and if Python tried to put any deadline on the workarounds, the only effect would be to place them between the hammer (Python) and the anvil (Apple). [Reply to this comment] Alex's approach Posted Jun 27, 2024 18:14 UTC (Thu) by mb (subscriber, #50428) [Link] (4 responses) >The question is, why would Apple have any incentive to fix it? I don't know. Apple *clearly* caused the breakage. That's why they should care. If they now decide to *voluntarily* drop Python support from the App store, it is their decision and we should respect that. If they care about Python on their platform, they can add an exception to their obviously completely broken checker. >So the odds are completely stacked against the app developers That's the case anyway. The Apple platform is fully proprietary and Apple decides everything anyway. And developers and users already agree to that. That's the deal. [Reply to this comment] Alex's approach Posted Jun 27, 2024 18:59 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses) > Apple *clearly* caused the breakage. That's why they should care. Why would Apple care about squishing an ant as they walk? [Reply to this comment] Alex's approach Posted Jun 27, 2024 19:04 UTC (Thu) by mb (subscriber, #50428) [Link] (2 responses) It's their decision. No (apps with) Python for Apple users then. I don't see why this is Python's problem. [Reply to this comment] Alex's approach Posted Jun 27, 2024 19:55 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (1 responses) Reality is that it's Python's users' problem, and projects have a tendency to listen to their users' problems, and do something about them. [Reply to this comment] Alex's approach Posted Jun 27, 2024 20:15 UTC (Thu) by mb (subscriber, #50428) [Link] Sure. Many Projects tend to put workarounds everywhere instead of really fixing things. Happy whack a mole. Users must complain to Apple. They broke Python. [Reply to this comment] Apple broke it Posted Jun 27, 2024 17:38 UTC (Thu) by mb (subscriber, #50428) [Link] Apple broke it, Apple ought to fix it. It's that simple. [Reply to this comment] Copyright (c) 2024, Eklektix, Inc. Comments and public postings are copyrighted by their creators. Linux is a registered trademark of Linus Torvalds