https://github.com/aws/aws-cli/issues/4947 Skip to content Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Issues - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} aws / aws-cli * Notifications * Star 11.3k * Fork 2.7k * Code * Issues 512 * Pull requests 197 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights New issue Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community. Pick a username [ ] Email Address [ ] Password [ ] [ ] Sign up for GitHub By clicking "Sign up for GitHub", you agree to our terms of service and privacy statement. We'll occasionally send you account related emails. Already on GitHub? Sign in to your account Jump to bottom publish v2 to PyPI #4947 Open mattsb42-aws opened this issue Feb 11, 2020 * 74 comments Open publish v2 to PyPI #4947 mattsb42-aws opened this issue Feb 11, 2020 * 74 comments Labels feature-request installation v2 Comments @mattsb42-aws Copy link Member @mattsb42-aws mattsb42-aws commented Feb 11, 2020 It's great that there's a first-class supported installation mechanism that does not rely on environment specifics, but this is still a normal Python package, and for those customers who do already have an existing compatible environment the installation mechanism is a lot heavier-weight than using the existing Python distribution mechanisms. Could we get v2 publishing to PyPI so that those customers who have those prerequisites can have a simpler and easier to maintain installation/updating/etc story? The text was updated successfully, but these errors were encountered: 162 2 We are unable to convert the task to an issue at this time. Please try again. The issue was successfully created but we are unable to update the comment at this time. @joguSD Copy link Contributor @joguSD joguSD commented Feb 11, 2020 Not publishing CLI v2 to PyPI has been a conscious decision and for the time being we have no plans to publish to PyPI. In the long term we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature. 5 228 39 6 @joguSD joguSD added the closing-soon label Feb 11, 2020 @sbbowers Copy link @sbbowers sbbowers commented Feb 12, 2020 @joguSD Could you please elaborate on the reasoning behind this conscious decision? Perhaps link us to some dialog around this discussion? 13 @ssbarnea Copy link @ssbarnea ssbarnea commented Feb 13, 2020 I think that time may prove if that decision was more towards conscious or in-conscious one. I am really curious to see how this will evolve, I only hope I have enough popcorn . On the other hand, other wheels coming from the same source did not get their notoriety without any reasons. boto, boto3 anyone? At least pypi was 3rd party that guaranteed that a specific version is not altered once uploaded. I guess now we will have to keep saving checksums . 7 @mattsb42-aws Copy link Member Author @mattsb42-aws mattsb42-aws commented Feb 13, 2020 * edited So pip install git+git://github.com/aws/aws-cli.git#2.0.0 then. :) 22 @no-response no-response bot removed the closing-soon label Feb 13, 2020 @ssbarnea Copy link @ssbarnea ssbarnea commented Feb 13, 2020 Sure, we all know that is impossible to force push code to github, not. 2 @adamchainz Copy link @adamchainz adamchainz commented Feb 13, 2020 Or use a tarballs link for greater speed https://adamj.eu/tech/2019/ 03/11/pip-install-from-a-git-repository/ 1 @joguSD Copy link Contributor @joguSD joguSD commented Feb 14, 2020 Just as a reminder, the AWS CLI V1 will continue to receive regular updates that are published to PyPI and pip will continue to be a supported installation mechanism for AWS CLI V1. Here are some reasons why we dropped publishing to PyPI for AWS CLI V2, and are not supporting pip as an official installation mechanism. 1. Installing with pip requires Python knowledge. The AWS CLI isn't a tool exclusive to Python developers and it's very easy to end up with an invalid installation. Often, reported issues end up being due to a misconfigured pip install that's getting dependencies from the wrong place, etc. These issues happen unexpectedly and are hard to debug which is frustrating for customers. It's also possible to inadvertently break other packages in the same environment. 2. Depending on a C extension can mean that at pip install time a C compiler and any other required libraries may need to be present for the install to succeed. Because of this we've avoided depending on libraries that require C extensions as having a compiler wasn't previously a requirement to install the AWS CLI. We also do not feel comfortable adding a compiler as a requirement to install the AWS CLI V2. For V1 this has limited our ability to implement some features as there are no suitable Python native libraries that meet our requirements (cryptography, etc). 3. Supporting pip as an installation mechanism requires supporting large dependency ranges for the packages we depend on. While simple in concept, this is difficult to execute on reliably as it's not feasible to test all permutations of platform, Python version, and package version. When new packages we depend on are published V1 customers installing via pip are often broken. This leads us to have relatively aggressive version ranges and we end up becoming a blocker for the Python community to upgrade packages. See #4749 #4561 #3535 4. There are some things we have no control of when installing via pip. For example, we have no influence on the openssl version or what TLS related features are supported as those will depend on how Python was built. This makes it difficult to ensure functionality of the AWS CLI. 5. We cannot include or ship additional binaries (that may not even be written in Python) as part of an AWS CLI install. In the future the AWS CLI can be partially or even entirely rewritten in another language while maintaining backwards compatibility. Hopefully that helps to shed some light on why we decided not to publish the AWS CLI V2 to PyPI. Lastly, we're looking to continue to improve the installation process for AWS CLI V2 by potentially integrating with homebrew and Linux package managers, and providing docker images. We'll also be looking to include build instructions for cases where the pre-built binaries aren't available or suitable for a particular use case. 27 89 9 3 @ulidtko Copy link @ulidtko ulidtko commented Feb 14, 2020 * edited Those sound as decent and understandable reasons -- except perhaps the #1. I'd avoid going into language comparison contests here -- but it's hard to refute that Python is among the few must know languages for DevOps folks and the like. Likely, I'm being ignorant in this judgement; but one can be forgiven for not "really knowing" even Bash -- but not Python. It's the language of humanity's first ever photo of a [shadow of] black hole, remember? --------------------------------------------------------------------- [DEL:Besides: that's some really wierd reasoning, don't you think? :DEL] * [DEL:Some users have hard time with pip, while others (most?..) pip users are quite happy and silent.:DEL] * [DEL:Yet others have hard time with prebuilt binaries, while others (most?..) binariy users are quite happy and silent.:DEL] * [DEL:Therefore, we'd punish ALL pip users, including the happy& silent ones, and force them to go use binaries.:DEL] Edit: sorry, disregard this. --------------------------------------------------------------------- Lastly: pip install https://github.com/aws/aws-cli/archive/ 2.0.0.tar.gz fails with: No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0) What's the route to go to install aws-cli v2 from source? 2 3 @quiver Copy link Contributor @quiver quiver commented Feb 14, 2020 @ulidtko Not only aws-cli but also botocore(low-level API) is not published to PyPI for v2, so you need to install botocore beforehand. For example, on a vanilla A1(ARM-base) Amazon Linux 2 instance, you can install AWS-CLI v2 as follows: $ sudo yum install -y python3 python3-devel gcc libffi-devel openssl-devel $ python3 -mpip install https://github.com/boto/botocore/archive/v2.tar.gz $ python3 -mpip install https://github.com/aws/aws-cli/archive/v2.tar.gz $ aws --version aws-cli/2.0.0 Python/3.7.4 Linux/4.14.165-131.185.amzn2.aarch64 botocore/2.0.0dev4 7 2 @thimslugga Copy link @thimslugga thimslugga commented Feb 14, 2020 * edited What about providing an AppImage for Linux vs the zip archive with the installer? I created one using the contents from the dist dir and it seems to be working fine. Just throwing it out there as it may be worthwhile way to distribute the v2 release for Linux. I can share steps I used if there's any interest. 2 1 @bristea Copy link @bristea bristea commented Feb 15, 2020 That like is like gold No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0) Translates to "We made a release using unreleased dependencies." and "We didn't test if the release artifact can be installed". Nobody mentioned is that binary code downloaded needs to signed on MacOS, so which means that the so called brew alternative install method will take considerable amount of effort/time. Let's face it, v2 was released even without a tag being made of github, this should say a lot. AFAIK it could also be made from another repository, another license. The v2 was not presented as a preview and was directly published on https://docs.aws.amazon.com/cli /latest/userguide/install-cliv2.html as a recommended release when the code repository, didn't had any tag, prerelease or not. I would think twice before arguing that so many users are not able to use pip, or compile code, when basic software development practices like tagging are not followed. 10 7 @sbbowers Copy link @sbbowers sbbowers commented Feb 17, 2020 Installing with pip requires Python knowledge... I agree. Doing things in a domain requires domain knowledge. Depending on a C extension can mean that at pip install time a C compiler and any other required libraries may need to be present... This is more domain knowledge, sure. Supporting pip as an installation mechanism requires supporting... It seems to me that this is your crux. Supporting PIP costs time and money and it sounds like it is generally painful. I suspect this is because, of the people that need support, they don't have the domain knowledge you'd mentioned. There is a difference between "supporting PIP as the method of install" vs "uploading your package to PyPi". You can keep on delivering your curl/unzip/bash method of deployment as official and supported and also allow, for the people that know and want to, install v2 via pip as an "unsupported" option. There are some things we have no control of when installing via pip... You may not have control over it, but you can check for it and make it dependent on a specific version. We cannot include or ship additional binaries... Additional binaries... sounds like dependencies, which could be checked for during the build process. The take-away I'm trying to make here is that supplying your packages via PIP isn't antithetical to your desire to limit support to your official installer. Would you please reconsider your aversion to pypi? Many developers would appreciate it. 21 [?] 20 @ulidtko Copy link @ulidtko ulidtko commented Feb 17, 2020 There are some things we have no control of when installing via pip. For example, we have no influence on the openssl version or what TLS related features are supported as those will depend on how Python was built. This makes it difficult to ensure functionality of the AWS CLI. Don't you think that this is quite intentional?.. To a user of your package, that's an atrocious idea: that installing an API client tool gives control over openssl on my system to the tool author. I'm getting shivers just from thinking of it! Constructively, there're two things you can do: * outright fail pip build, if the given openssl isn't good enough; * graceful degradation, where parts of functionality which depend on missing TLS features just get disabled. --------------------------------------------------------------------- We cannot include or ship additional binaries (that may not even be written in Python) as part of an AWS CLI install. In the future the AWS CLI can be partially or even entirely rewritten in another language while maintaining backwards compatibility. Why, what's wrong with setup(data_files=...) and/or MANIFEST.in, did you look into it? Thousands of developers do include arbitrary files in their pip installs. I see no problem here. Oh wait, actually I do. Binaries don't belong in source installs. If I'd want to rewrite aws-cli in golang or something -- I'd make it separate project, in a separate repo, and maintain its interface as "CLI compatible" rather than vaguely "backwards-compatible". Don't ship binaries where source is expected. No-no. No no no no. 23 1 @dragon788 dragon788 mentioned this issue Mar 3, 2020 Missing version download instructions #5013 Closed @benbridts Copy link @benbridts benbridts commented Mar 9, 2020 * edited @joguSD Would it make sense to have separate issue for "install with a package manager" or "install with package manager X". Currently It's hard to automate the installation with the bundled installer (because you may not have a directory in your PATH that's suited for not-packaged binaries). And it could be a good place to consolidate all the related issues like this one, #4926 #4842 #4660 #3553 #3627 #2501 and #727. With v1 the suggestion was (probably rightly so) to use pip when needing a packaged/automated installation, with v2 this isn't an option anymore 1 @allenatdecisiv Copy link @allenatdecisiv allenatdecisiv commented Mar 17, 2020 @joguSD what about platforms you don't directly support but language packaging mechanisms (like PyPi) do? For instance, Cygwin? 1 @jpetazzo Copy link @jpetazzo jpetazzo commented May 3, 2020 Hi! I do understand that AWS is trying to make it easier to install the AWS CLI for users who are not familiar with Python. It is regrettable that it is done at the expense of users who are familiar with Python, in a way that prevents them from integrating the AWS CLI within many environments. For instance, many platforms will let you specify Python dependencies in a requirements.txt or similar file. A package published on PyPI becomes installable on these platforms without any extra or special work. A package installed through a script that requires root does not. This is the case on Heroku, and it seems to be the case on AWS Lambda too. If I want to run a script using the AWS CLI on Lambda, it looks like I could do it with AWS CLI v1 (since the Lambda documentation explains how to package Python dependencies) but I don't know if it'd be possible with AWS CLI v2. There are also many platforms that won't allow root access. A package published on PyPI can be installed as an unprivileged user, while the AWS CLI v2 installer requires sudo. There are ways to have your cake and eat it too: check out Docker Compose installation instructions. Compose is available as a standalone executable for most platforms, and can also be installed with pip install. I understand the desire to make Python an "implementation detail". But every software is written in some language, and that language has best practices around installation procedures. For instance, C code can leverage e.g. autoconf and automake, and every Linux distribution will have helpers to facilitate packaging software that uses autoconf and automake. Likewise, software written in Python can leverage PyPI. Finally, if CLI installation is truly a huge point of friction, there are more drastic solutions as well. For instance, it could be ported to Go, to allow producing a standalone binary for every major platform. In 2012, I was working at Docker Inc. when we made the decision to rewrite our container engine from Python to Go, in order to facilitate its adoption by the largest number of people. I do understand that the AWS CLI is a complex piece of software that needs to interface with hundreds of services. But if a 15-person start-up (the headcount of Docker back then) can rewrite a container engine in Go, perhaps a company like AWS can manage to port a CLI as well? ;-) If porting the CLI to Go seems to big a burden, I'd suggest to at least offer a PyPI installation process. It's fine if it's "not officially supported". At least don't make our job harder. Thanks, 33 [?] 7 @titan550 Copy link @titan550 titan550 commented May 3, 2020 The installation instructions look like a alpha release note. However, the documentation suggests that v2 is a prod release. The only installation method that seems to be prod ready is Windows MSI (I have not tested). Please provide options for installing using package managers such as pip etc ASAP. 7 @borrell Copy link @borrell borrell commented May 4, 2020 Unfortunately a new core AWS feature (SSO) is a non-starter for us for the moment, until 1. CLIv2 is available through some sort of versioned repo (pip, distro repos, anything) and not just an installer linked on an AWS documentation page 2. Core SDK support for basic things like 'use SSO to authenticate' are available (this is purely directed at botocore). 8 @tmccombs Copy link @tmccombs tmccombs commented Jun 2, 2020 Another problem with the current install process is that while there are instructions on how to update the package, there doesn't appear to be any documented way to programmatically check if an update is available. 3 @bendog Copy link @bendog bendog commented Jun 5, 2020 I'm having to use the new aws sso login feature a lot, and since i'm working inside virtual environments, which use tools which have awscli as a dependency, I have to exit my virtual environment every time i need to use the sso login (hourly). Please save me from this! please release to PyPI!! 6 @vrdhn Copy link @vrdhn vrdhn commented Jun 15, 2020 * edited For my own future reference pip install --user https://github.com/boto/botocore/archive/v2.zip https://github.com/aws/aws-cli/archive/v2.zip ## Not working anymore ... pip install --user git+git://github.com/boto/botocore.git#v2 git+git://github.com/aws/aws-cli.git#v2 [?] 16 @jsumners jsumners mentioned this issue Jun 16, 2020 When will version 1 be EOL? #5294 Open @mcfedr Copy link @mcfedr mcfedr commented Jun 17, 2020 Another disadvantage with not publishing to pip, you making me have a second python interpreter installed, which might be fine for a dev machine, but in a docker image isnt good. This is related to the issues installing on alpine linux in #4685 - if it just used the system python (as in v1) then would be no problem. 8 tim-bm added a commit to tim-bm/kube-tools that referenced this issue Jul 28, 2020 @tim-bm add aws cli support ... Verified This commit was created on GitHub.com and signed with GitHub's verified signature. GPG key ID: 4AEE18F83AFDEB23 Learn about vigilant mode. 9fe85c1 The reason for installing version one aws/aws-cli#4947 @tim-bm tim-bm mentioned this issue Jul 28, 2020 add aws cli support Place1/kube-tools#4 Open @NewbiZ Copy link @NewbiZ NewbiZ commented Aug 11, 2020 * edited Installing with pip requires Python knowledge. The AWS CLI isn't a tool exclusive to Python developers and it's very easy to end up with an invalid installation. Often, reported issues end up being due to a misconfigured pip install that's getting dependencies from the wrong place, etc. These issues happen unexpectedly and are hard to debug which is frustrating for customers. It's also possible to inadvertently break other packages in the same environment. Not sure I get your point here. Installing with pip does not mean pip has to be the only way to install aws cli. e.g. There are already RPM and DEB packages of AWS CLI at the moment. Depending on a C extension can mean that at pip install time a C compiler and any other required libraries may need to be present for the install to succeed. Because of this we've avoided depending on libraries that require C extensions as having a compiler wasn't previously a requirement to install the AWS CLI. We also do not feel comfortable adding a compiler as a requirement to install the AWS CLI V2. For V1 this has limited our ability to implement some features as there are no suitable Python native libraries that meet our requirements (cryptography, etc). Again, I do not get the point you are making here. Nobody is forcing AWS CLI to be distributed as a source python package. You can ship pre-built versions of AWS CLI as wheels to pypi, thus have no build-time requirement during installation. There are some things we have no control of when installing via pip. For example, we have no influence on the openssl version or what TLS related features are supported as those will depend on how Python was built. This makes it difficult to ensure functionality of the AWS CLI. What are you implying here? That you will now ship your own python interpreter and openssl inside your new AWS CLI "bundle"? Considering your main point seem to be the support burden, I fear this decision will just change your support requests from "help me aws cli is broken because of my openssl version" to "please release a new bundle with an updated openssl because of CVE-42". Will I have to independently update AWS CLI whenever there is a security fix in python/openssl ? We cannot include or ship additional binaries (that may not even be written in Python) as part of an AWS CLI install. In the future the AWS CLI can be partially or even entirely rewritten in another language while maintaining backwards compatibility. Of course you can, multiple libraries on pypi do ship with prebuilt dependencies embedded. You can either statically link your bindings with the depended library during wheel creation, or just directly embed the shared/ dynamic library in your package and use that for your cffi calls, there's plenty of ways to do that. --------------------------------------------------------------------- To be honest, this move of getting out of pypi is really getting on my nerve. The code quality of boto3 and AWS CLI is abysmal, and all I see here is a escape plan from that codebase. I am pretty sure we will soon have an announcement along the lines of "Sorry, we screwed up in Python, we are now working on porting AWS CLI to Go, we hope it's going to be better <3" 23 1 [?] 12 @tmccombs Copy link @tmccombs tmccombs commented Aug 11, 2020 Considering your main point seem to be the support burden, I fear this decision will just change your support requests from "help me aws cli is broken because of my openssl version" to "please release a new bundle with an updated openssl because of CVE-42". This is especially bad, since there is not a great way to automate upgrading the AWS CLI now. 38 hidden items Load more... @kojiromike Copy link @kojiromike kojiromike commented Jan 26, 2021 You can load zipfile as a module on the command line This is a great tip, but nevertheless misses the point of this issue. pip install should be a supported way to install awscli2. 6 @ackalker Copy link @ackalker ackalker commented Jan 27, 2021 * edited You can load zipfile as a module on the command line This is a great tip, but nevertheless misses the point of this issue. pip > install should be a supported way to install awscli2. Of course you are right, my comment was meant tongue in cheek. No project should force its users to download and install an entire separate Python environment built from who knows what sources just to run a single tool. There are shops out there that have very strict rules about what software is allowed to run on devops' machines, usually the need to vet the source code from which that software is built, and no random binary blobs downloaded from the internet, even if they are hatched by Amazon, are ever going to fit that bill. 5 @henning-schild Copy link @henning-schild henning-schild commented Jan 27, 2021 If you want to sell your services you better bend over backwards to make those tools readily available in a professional manner. Meaning packages in all Linux distros ... skip pypi if you want but things like https://bugs.gentoo.org/767268#c1 are shocking results of a really poor deployment channel for that essential component. sudo funny-script-without-signature-or-checksum ... 8 @danpilch Copy link @danpilch danpilch commented Feb 8, 2021 * edited Removing the path of least resistance (installing via pip) just makes me not want to upgrade. I'm sure a lot of other people will ignore v2 until v1 is EOL and we're forced to install a python cli tool in a non native way. 5 @ulidtko Copy link @ulidtko ulidtko commented Feb 8, 2021 I'm sure a lot of other people will ignore v2 until v1 is EOL I'm sorry if this is bitter feedback for some, but having learned that AWS staff has difficulties with basic packaging and versioning so much that they want to make Python an implementation detail of aws-cli -- I'm reluctant to use even v1. Who knows what other nonsense they decided to put into there; I don't care enough to review the 70kSLoC myself. 7 [?] 1 @kdaily kdaily mentioned this issue Feb 9, 2021 Improve documentation of versioning and releases for the Python SDK and tools ecosystem boto/botocore#2295 Open @tboerger tboerger mentioned this issue Feb 10, 2021 Added AWS codecommit support using access & secret keys drone/ drone-git#36 Merged @danvaida Copy link @danvaida danvaida commented Feb 12, 2021 $ wget https://awscli.amazonaws.com/AWSCLIV2-2.1.26.pkg $ open AWSCLIV2-2.1.26.pkg $ aws --version aws-cli/2.1.26 Python/3.7.4 Darwin/20.3.0 exe/x86_64 prompt/off Meanwhile, a pip-installed CLIv1 runs fine on Python 3.9. Now, I think therein lies a tale... @vacri Copy link @vacri vacri commented Feb 18, 2021 I'm clearly late to this conversation... but just to highlight a major issue again: package managers are there for a reason. Throwing away package management altogether is such an incredible antipattern. There is so much that package managers do in the ops space. Even in the environment of the dev workstation, where there's a human to control everything, what used to be [package manager] install [package] is now 1. hunt for webpage with link 2. run a command to pull the package down 3. find a temp dir to extract it to 4. scan the install script because it's a custom script doing a custom thing and who knows what it might do - hope you're familiar with shell! 5. run the install 6. test it That's the recommended "happy path" with no edge cases. And then repeat that all again when there's an update... if you remember to check for one. This is not the right method for less-skilled admins, plus rather than "doing the work once" by the developers, you're making every admin duplicate the work. If not pip because you want to go language agnostic, then why not rpm for Amazon Linux and maybe deb because it's the other super-popular package format? Both of these are designed for dependency management and deal with upgrades fluidly, and you don't have to hunt for a link in a webpage for packages in them. I don't have the venv or docker problems that other users have but I'll keep using v1 anyway, because updating to get new features is a one-liner, rather than a manual zipfile install method from last century (no joke there - rpm's from '97, apt's from '98, cpan '95...) 12 @danizen Copy link @danizen danizen commented Feb 20, 2021 The reason this is an anti-pattern for development is that a different version of botocore may be installed for other reasons, and so we Python developers who use virtual environments simply cannot use awscli v2 without using the workarounds above, because it would conflict with the packages used by other tools. This works against proper development practices. Some community work-arounds, such as pypi package awscliv2, work for me only at home because within my federal agency, we do not yet have access to Docker or WSL, and python virtual environments are my way to manage all dependency isolation. 7 @ShadowJonathan Copy link @ShadowJonathan ShadowJonathan commented Feb 21, 2021 @danizen i would recommend keeping working with venvs, because they're so widely used, and so more manageable than complete system environments, just a note. @mvanbaak Copy link @mvanbaak mvanbaak commented Feb 21, 2021 we do not yet have access to Docker or WSL, and python virtual environments are my way to manage all dependency isolation. Also, I dont want to install and run docker just for the awscli. It's not that I cant install it, it's just that I dont use docker, and having to go that route just for a cli tool ... yeah, that's a NOPE @slashbeast Copy link @slashbeast slashbeast commented Feb 21, 2021 Here you have a very well done paper on why the route you've choose with awscli v2 is wrong -- https://blogs.gentoo.org/mgorny/2021/02/19 /the-modern-packagers-security-nightmare/ 4 [?] 1 @zerthimon Copy link @zerthimon zerthimon commented Feb 21, 2021 Unbelievable. And the worst part is - it's Amazons official. I wouldn't be surprised if it was Alibaba cloud or something, but it's Amazon. Really ? Wow, just wow. 2 @ShadowJonathan Copy link @ShadowJonathan ShadowJonathan commented Feb 22, 2021 This issue isn't going to progress anything, it's not getting closed nor resolved, but kept open to blackhole discussion, nothing will come from this, and nobody can force amazon to do anything about it. So i suggest finding other ways to forward this issue, other than leaving behind a comment here saying your support. I've been watching this issue thread for a while now, and I haven't seen a member respond to the latest concerns and calls for python best-practice here, other than suggesting an entirely un-ergonomic way of deployment, not even willing to meet the developers half-way, dismissing releasing on pypi in favour of a vendored appimage blob. Earlier here it's already been said that amazon considers using python in general as an implementation detail to how the aws cli is run, fine, fair, but then it acts as if it's above any choice it makes in that regard, e.g. not answering to what the community of python expects of their standardised way of publishing a package for a 30-year-old programming language, on a 20-year-old publication site, for a cloud service that has then only existed for 15 years, and then act as if their way is the highway. Well then, @aws, fucking commit, make your own https://pypi.aws instance, dont just sit around and dabble with this absolute bullshit. You love developers so much? You love your customers so much? Prove it then, your platform has been a hot piece of overpriced garbage for a decade and a half now, so you cant change my mind, but you might change others. @ulidtko Copy link @ulidtko ulidtko commented Feb 22, 2021 Just boycott awscli v2. Don't use it. If anyone ever wonders why -- explain. Hopefully you can be understood. @bhanutejags bhanutejags mentioned this issue Feb 24, 2021 Build the docker containers in CircleCI uclibs/treatment_database#128 Closed @danizen Copy link @danizen danizen commented Feb 25, 2021 I do see the problem with installing in a virtual environment from AWS perspective - there can be as many installations of AWS CLI as there are virtual environments, and together with user knowledge and the issue of including other capabilities maybe not written in Python and controlling the version of SSL (LibreSSL probably on their horizon), it makes sense. However, I have programs that use boto3 to do things similar to aws sso login, and they rely on boto3 and therefore botocore. Bottom-lines: * If it cannot be published to PyPI then you need to assure that when I invoke the command-line aws from within a virtual environment, it uses its own version of botocore. I will be testing this and reporting back with V2 on Windows and then on Mac. * It has to be published in a package format where we can programmatically determine updates. This is part of Software Supply Chain management for developers & DevOps, but also part of IT Asset Management, Users should be able to programmatically determine whether there is a new update available. * It needs to be installable without administrative privileges on Windows, due to the many users who use AWS CLI who do not have administrative privileges to their own computer. Git for Windows does this - it is not that hard. If these three requirements can be met by AWS, I see no problem with the lack of PyPI. Making an image available addresses the first two requirements, but the 3rd requirement is not met because many Windows users still are not allowed to use WSL. @tmccombs Copy link @tmccombs tmccombs commented Feb 25, 2021 It has to be published in a package format where we can programmatically determine updates. This is part of Software Supply Chain management for developers & DevOps, but also part of IT Asset Management, Users should be able to programmatically determine whether there is a new update available. This is this biggest thing for me. I'd actually prefer having it in an apt repo to PyPi, but the current only official deployment mechanism is... not good. I also don't love that an entire python runtime is bundled with it. Or that if you upgrade the aws-cli, it leaves the old version lying around. @tmccombs tmccombs mentioned this issue Feb 25, 2021 V2 Update leaves old versions behind #5979 Open 2 tasks done @zerthimon Copy link @zerthimon zerthimon commented Feb 25, 2021 * edited Well, we are at 2021 and rely on a zip file and bash script with couple cps and lns to install python libraries, how cool is that ? Seriously, I now need to figure out how to install it with a SaltStack on all my ec2 instances, making it idempotent and upgradable. Thanks for thinking about making it easier for DevOps. 4 @boltronics Copy link @boltronics boltronics commented Feb 26, 2021 It's so easy to upgrade awscli v1. pipx upgrade awscli Done. No root privileges required. It's what all the paying AWS customers want. Instead we get this 20 step process which requires a dedicated installation page to explain (instead of a couple lines of the README file) which discourages upgrading. This probably creates more of a risk than anything else honestly. I too am probably going to want to write a Salt state to manage this, but how many AWS customers are going to bother with that? Shouldn't AWS be taking on the inconveniences themselves instead of trying to push them onto paying customers? Installing with pip requires Python knowledge. Then continue to provide the already existing installation methods too. Depending on a C extension can mean that at pip install time a C compiler and any other required libraries may need to be present for the install to succeed. If you're already supplying another installation method, what does it matter? It's the user's decision. Supporting pip as an installation mechanism requires supporting large dependency ranges for the packages we depend on.... See # 4749 #4561 #3535 Just say you won't support anyone who isn't using a dedicated virtualenv environment for awscli. That would prevent you from dealing with all of those issues. There are some things we have no control of when installing via pip. For example, we have no influence on the openssl version or what TLS related features are supported as those will depend on how Python was built. Yes but it's not that hard to check and fail installation if your requirements are not met, with the appropriate error message. I think most users wouldn't have a problem, unless you decide to rely on some bleeding edge library feature (which wouldn't be a great idea anyway). We cannot include or ship additional binaries (that may not even be written in Python) as part of an AWS CLI install. Then have awscli initiate a check for whatever dependencies there might be (if ever that becomes a thing) at runtime where required. It's not a big deal. But even if it was, this isn't your typical free software project where people can only hope, beg and perhaps offer a donation to help persuade a change in how things are done. In this case, you are getting paid to write this library because we are all paying for the AWS service. Hence I would expect AWS developers to take on some of the inconvenience to satisfy customers. Well that's my thoughts on the matter. I'm a bit frustrated because I went to use aws ecr get-login as per the docs it wouldn't work, and I had to investigate why pipx wasn't upgrading my awscli setup, until I stumbled upon this huge issue. 8 @ShadowJonathan Copy link @ShadowJonathan ShadowJonathan commented Feb 26, 2021 * edited @joguSD any response yet? Or are y'all explicitly ignoring this issue, even when it's attracting a lot of attention (even today) because you're ignoring one of the oldest language's best practices? Do you all really think you're so above that to not bother with these "silly practices"? I re-read your first message, if you're not going to use python for awscli, commit, don't fuck around and say "oh but we might not do that", it's been a year, people are still annoyed at this not being on pypi, put it on pypi. @ck3mp3r Copy link @ck3mp3r ck3mp3r commented Feb 26, 2021 Ironically I can see a few of the Amazon leadership principles having been neglected and even broken with how they treat this issue. For one I don't see "Customer Obsession" being respected with this approach... 6 @kellertk Copy link Contributor @kellertk kellertk commented Feb 26, 2021 Hi everyone. First off, thank you all for the frank feedback about this decision, particularly specific feedback from some on this issue about why this matters to you. While @joguSD has already detailed why the team decided to not support pip, the continuing feedback is very important to us and we do use these anecdotes in future planning. However, I would like to ask that we try to keep this conversation civil and constructive. Just as a friendly reminder, contributions to this repository, including issue comments, are governed by the Amazon Open Source Code of Conduct. For our part, I can promise that we're reconsidering this decision internally based on the feedback we've received here. We want to consider the implications of a such a change (see @joguSD's comment linked above), so that will take some time. Expect an update from the team on that front by the end of next week. 11 6 [?] 7 1 @ericfrederich Copy link @ericfrederich ericfrederich commented Mar 4, 2021 * edited The aws cli is not without bugs. There have been times I've had to step my way through a command in a debugger like PyCharm to find the issue. For this, it is very useful to be able to install via pip or pipx. Even if I clone this project and check out v2 branch I cannot install with pip install -e . because it's depending on some non-published botocore==2.0.0dev97 @allenatdecisiv Copy link @allenatdecisiv allenatdecisiv commented Mar 4, 2021 Even if I clone this project and check out v2 branch I cannot install with pip install -e . because it's depending on some non-published botocore==2.0.0dev97 @ericfrederich, I am guessing that is by design and intentional? They aren't tagging v2.x.x releases in botocore. To find the "2.0.0dev97 release" (ahem) of botocore you must read the commit log messages from the v2 branch. Then check out out the commit. Then build a wheel like python setup.py bdist_wheel. Then install or upgrade via pip (with the corresponding awscli v2 wheel). YMMV... @michael-k michael-k mentioned this issue Mar 5, 2021 Wheels are only universal if they support Python 2 & 3 #5987 Open @shaunhickson Copy link @shaunhickson shaunhickson commented Mar 9, 2021 Hi all - I'm the manager of the AWS CLI team, and I wanted to address the conversation that's been going on here, as well as some broader issues. I appreciate everyone who's taken the time to comment. Every comment on every issue in this repository is read by multiple members of the team, and we value every anecdote and data point about how our product works or doesn't work for you. One of our goals for the rest of 2021 is to do a better job of making that communication bi-directional and to more directly engage with the community on longer-term plans and roadmaps. I'll discuss some of our plans for that, but first I want to address the issue at hand. We do not plan to publish the AWS CLI project to PyPI at this time. I appreciate that this will not be a popular decision among people following this issue. In Q2, we plan to begin work on a source bundle that is well documented, easily ingestible by package managers, and that can be used to build the CLI for platforms other than those we support out of the box. This source distribution will include botocore along with the CLI. Our hope is that providing an officially-supported source distribution will alleviate many of the frustrations that you have experienced with our current approach. We understand that we need to make our product easily available for those who can't or don't want to use one of our pre-packaged binaries or container images. Once we've released and validated the source bundle, we'll open an issue here to prioritize distribution channels that will provide the most value and ensure that maintainers have what they need from us. Starting in April, we're going to begin posting monthly roadmap updates as issues on our GitHub repository. These will lay out what we're actively working on and what's on the horizon. Our backlog includes a mix of priorities, some driven by the CLI team and the community, some which align with larger AWS goals. Our focus will be on sharing the priorities emerging from our team, which will include Github issues we are planning to work on. We will iterate on the exact format and approach for these updates as we determine what works best, but they will be at least monthly for the rest of 2021. I'm going to lock this issue now, and we will be closing any new iterations of it as duplicates. I would ask that everyone please keep in mind the Code of Conduct https://aws.github.io/code-of-conduct when using GitHub to communicate with the team. @aws aws locked and limited conversation to collaborators Mar 9, 2021 Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in. Assignees No one assigned Labels feature-request installation v2 Projects None yet Milestone No milestone Linked pull requests Successfully merging a pull request may close this issue. None yet 43 participants @kdaily @ssbarnea @danvaida @jpetazzo @slashbeast @borrell @vemel @NewbiZ @boltronics @quiver @thimslugga @sbbowers @ulidtko @ackalker @pcgeek86 @ck3mp3r @mcfedr @vacri @zerthimon @adamchainz and others * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.