Labels versus Stages

We've been discussing this already in https://github.com/iterative/gto/issues/68
Also we recently got a user request of adding Labels mechanics in #179
Feedback
I've been asking a couple of question about Stages vs Labels approach in multiple places:

In ods.ai Slack I got 5 votes for Labels and 1 vote for Stages.

In MLOps.community I got this feedback

Suggestion
Considering answers, I think we need to either

Change Stages to Labels (and assume users will implement Stages mechanics by themselves if they want to) (also we have labels in artifacts.yaml already - how to fix this?)
Support both. In this case git tags for Stages could be denoted as rf:prod (giving a hint about dicts and 1:1 relation), while Labels could be denoted as rf#prod.

WDYT @omesser @dmpetrov ?

@aguschin  - from the responses I think we can infer that the labels mechanism is largely needed to be "declarative" (reflecting something about the model itself, or even about where it is already deployed), but not necessarily "actionable" - meaning:

stages - I would like the event of state-change to trigger workflows
labels/annotations - I would like the changing of labels/annotations to ONLY help with searching / filtering, not trigger anything

for controlling n:n rel with which model is deployed where (and be more flexible than stages) - we need to have a 3rd and separate concept of:

"deployments" or "environments" The reason I'm not connecting this with stages/labels(=annotations) is that it should be clear / intuitive that this is MEANT to trigger workflows / deployments / checks. We probably can't mix between a declarative mechanism like 2 and this - because people want something lightweight to tag stuff to filter on without triggering anything

What do you think?

Labels make more sense for my use case.  We have models that are shared between different types of hardware and we have a group of beta users that use the models before they go to production.  In this case I would like to be able to use gto to trigger a deploy to our beta users hardware, once QA and the beta test are complete I would like to use gto to trigger a deploy to production.  Using gto show I would like to see the current state of deployment - something like:

name
latest
#hw1_prod
#hw1_testers
#hw2_prod
#hw3_testers
#staging

model
v1.0.2
v1.0.1
v1.0.1
v1.0.0
v1.0.1
v1.0.2

When an engineer gets called on to investigate a system, they should know whether it is hardware 1 or hardware 2 and if the problem system is production or part of the beta test group.  I would like them to be able to quickly check what version of the model the system should have with gto, sometimes we investigate a problem system that may have failed to update to the proper model, gto could make that issue clear quickly.
I recognize the value in the stages approach, especially if QA is fully automated, the test hardware is only for testing (not a product being used by customers/employees), and only one production environment is supported for a model.

Thanks for the input @shortcipher3 !
This sounds like a clear #3 from my comment here -  the physical environment where the models are deployed to.
Note that the stages in the approach today are distinct from deployment environment/targets.
And we're trying to figure out whether we want to replace stages with labels, and how they should be used correctly, but the deployment environments themselves, in my opinion, are distinct from those either way and should be managed by the model deployment (like mlem) tool and not gto.
Take this scenario for example:
I have 2 models: nn1, nn2, and 3 production setups - setup1, setup2, setup3.
When I move a version of either of the models from development stage to integration stage:

there's a series of tests running against the model

Now, when I move some model-version (for either model) from integration stage to production stage:

there's a larger series of tests running against them, and once that's successful, triggers the next piece of logic
if model nn1 -> deploy on setup setup1 (because model nn1 is always deployed on setup setup1, the model is built for the app running on this setup)
if model nn2 -> deploy it on setup setup2 +  setup3 (same thing, those setups are hosting the some instances of the app that works with nn2)

Notice there is an assumption here that the affinity between model and setup is "strong" - i.e. not "loose" / controlled by a human deciding to label a model this or that, because the models and the workflows were set-up to always deploy specific models to specific environments. modes<>envs are coupled.
In other words -
If such a strong affinity between model and setup indeed exists, than stage->setups should be codified in the workflow (the code that is triggered by promoting to stage X) and not by separate gto / model registry command - where a human chooses a setup manually.
What do you think? Does that make sense?

Thanks for the detailed answer, @shortcipher3! That's very interesting.

I'm probably missing some things from mlem or possibly there are gaps in mlem that I was hoping gto would address. I see how I can create the deployment environments, but I don't see how to check what is currently deployed to each deployment environment.

This is something we're going to add to MLEM this year. The idea is, if you have models deployed to some "setups" (MLEM Environments in this case), you'll have mlem deployment status or something that will poll all known deployments and output them with their status. Taking your example above, I guess the output will look like this:

name
latest
#hw1_prod
#hw1_testers
#hw2_prod
#hw3_testers
#staging

model
v1.0.2
v1.0.1 [running]
v1.0.1 [running]
v1.0.0 [failed]
v1.0.1 [running]
v1.0.2 [request timeout]

That said, this example currently extends to deploying model to Heroku, let's say. Not the scenario you describe.
Still, there are MLEM plugins, that can enable any kind of deployment actually. Let's suppose your deployment happens this way:

You train model somewhere (you don't use MLEM to save it)
You export your model to .onnx
You create git tag with GTO that supposed to deploy your model to some environment

If we implement MLEM plugin that could deploy .onnx to your specific env (and serve it with any framework you want I think) and poll the service to know everything is ok, then it would look like this:

and 2. are the same.
You import your .onnx model into MLEM
You deploy that model with MLEM to your specific env manually. Or set up CI that reacts on GTO promotions (implies strong affinity or some user-input though).

Did I get your case right? Does it look like an adequate solution?
TBH, I think this can work, but still looks like a complex solution if someone needs to just trigger deployments with git tags with Labels mechanism, and doesn't care about whether it's actually deployed or not, cause he's checking some other source for that information. cc @omesser

What I like about gto is I can tag models as well as arbitrary binaries (such as builds of the software that serves the models).  I like that it is purely a cli tool.  I really like gto isn't forcing me to use my models a particular way and that it isn't geared around deploying as a http endpoint.  I like that I can easily get a listing of things that I've tagged in a nice tabular format that includes a list of binaries and a list of stages/label.

MLEM can also be a CLI-only tool I think, if we publish it as a .deb package let's say. Will it work for you? What OS is running on your iot devices? What package manager do you use there?

I really like that I can create links between models in git projects in mlem.

Could you please share some feedback for this? How do you intend to use those?

@aguschin
Sounds like a dummy deployment implementation in mlem would be a bit artificial - the user will still have to introduce mlem to their workflow, seemingly without a good reason, and we will have a concept in mlem that doesn't make sense to people actually deploying models with mlem 🤔 .
I'm entertaining the idea of having an "environment" concept in GTO (In addition to mlem. Yes, overlapping concepts... 😭 ). Maybe we can come up with something simple that would provide a consistent GTO CLI experience e2e and would wrap around mlem environments as well (as extension) to some native gto simple implementation, to support an e2e gto workflow.
wdyt?

I don't like the overlapping concepts either, and I don't like having two mechanics in GTO, because it complicates things, but if we cannot support user scenarios with a single concept, I guess we need to have both.
A possible scenario, in this case, looks like this for me:

When you start working with GTO, you decide, what you want to use - Stages or Labels (I doubt if you need both in the majority of situations)
If you choose Labels, then in a naive approach, for each GTO Label you are assumed to have the same MLEM env
When you create a GTO Label (git tag) and push it to repo, CI starts
The naive approach allows you to parse the git tag and reuse it "as is". E.g. if it's "rf:prod:1", then you ask MLEM to deploy a MLEM model in path "rf" to a MLEM Env "prod"
Another level of indirection here (We let the user work with this indirection by himself.) is
a. if GTO model named "rf" have a different path in repo (e.g. "models/random-forest").
b. if GTO Label is called "prod", but MLEM env is called differently (e.g. "production"). Or GTO label "prod" matches to multiple MLEM envs (e.g. "prod-heroku", "prod-sagemaker").

The other question is what we will see in Studio. With this approach, it looks like eventually we would have to have "Stages", "Labels", and "Deployments" columns here?

Hi @shortcipher3! We're decided to introduce "labels" mechanics we've been discussing. I'm updating README and the code in this PR #218. If you could check it out and provide your feedback, that would be great!

@shortcipher3  - as you can maybe see from #218 we've been having some internal discussions on this. Now you can use the labels to represent environments/deployments without a dedicated environments concept, but you'll be responsible to unlabeling as well to make it behave logically - e.g.

you deploy model v1.0.0 on #hw1_testers -> you label v1.0.0 with #hw1_testers
you deploy model v1.0.1 on #hw1_testers -> you unlabel v1.0.0 #hw1_testers + you label v1.0.1 with #hw1_testers (to represent the overriding property of deploying on the same hw/set of hw targets)

This is a better fit for my use case, thanks for the updates.  I'll work with it and see how I like it.


Ng command only works when using sudo

Please provide us with the following information:

OS?

Windows 7, 8 or 10. Linux (which distribution). Mac OSX (Yosemite? El Capitan?)

MacOS Sierra 10.12.1
Versions.

Please run ng --version. If there's nothing outputted, please run in a Terminal: node --version and paste the result here:

sw_vers -productVersion
10.12.1
npm --version
3.10.3
node --version
v6.7.0
sudo ng --version
angular-cli: 1.0.0-beta.16
node: 6.7.0
os: darwin x64
Repro steps.

Was this an app that wasn't created using the CLI? What change did you do on your code? etc.

Here is a recording I made to show the issue
https://asciinema.org/a/aqnrpimuf0agzznrq5fp7acss

The log given by the failure.

Normally this include a stack trace and some more information.

Mention any other details that might be useful.
None

Thanks! We'll be in touch soon.

I experience the same problem. #2426

How long have you waited for it to start since in the video you kill it quite quickly? It will take some time when not ran as sudo. The command itself is answering since there is "command not found" or anything.

Dupe of https://github.com/angular/angular-cli/issues/2426, seems to have a solution in https://github.com/angular/angular-cli/issues/2426#issuecomment-251557545.


Fix handling of deferred fields on django 1.10+

Fixes #278. cc @utapyngo @djstein
Problem
Prior to Django 1.10, deferred attributes were handled by constructing a new model class with custom descriptors for the deferred attributes. After 1.10, deferred fields are tracked by whether the attribute is present in the instance __dict__.
FieldTracker handled tracking on these fields by overwriting the descriptors on the model class. This means that in 1.10+, the instance model class is the model, so FieldTracker can introduce changes to the base class whenever deferred fields are used that persist on other queries. #278 This is reproduced in a test case.
Solution
I updated the finalize_class method to wrap field descriptors. This preserves the custom descriptor behavior, and means that deferring fields does not lead to permanent changes in base model classes. This is only done for django versions after 1.10: the previous behavior is preserved for 1.8 and 1.9.
Since the relevant code branches behavior on unsupported versions of Django, I also removed all branching behavior for versions of django listed as unsupported in the CHANGELOG.
Commandments

[ ] Write PEP8 compliant code.
[x] Cover it with tests. Reproduced in test cases, which fail prior to 54cc1507a79460497d40f6cba779f67a4b5f8041 and pass afterwards.
[x] Update CHANGES.rst file to describe the changes, and quote according issue with GH-<issue_number>.
[x] Pay attention to backward compatibility, or if it breaks it, explain why. The only backwards incompatible change is that the tracker._deferred_fields attribute is not set on django versions >=1.10. Since _deferred_fields is a private attribute, I wouldn't consider this a breaking change.
[x] Update documentation (if relevant). _I don't think it is relevant.

Regarding PEP8, I ran flake8, and it generated a number of errors. I think my code didn't introduce new PEP8 violations, but I'm not sure. If you want, I can submit a separate PR fixing all the PEP8 violations, and rebase onto this branch.

@jcushman I think this now includes fixes from #313, updated to work on Django 1.10+. Could you take a look?
@carljm What is required to get this merged and deployed this to pypi?

OK, I got the codecov checks passing by correctly aggregating the different test environments and adding a few assertions to the tests.

Hi! I don't have my head super in this, but I see that my tests are there and passing, which is great. 👍
It sounds like you can join Jazzband and merge this yourself, if you're comfortable with the code at this point: https://github.com/jazzband/django-model-utils/pull/279#issuecomment-386069220
Alternatively you might @ a few folks who have merged PRs in the last year for another pair of eyes.

(Oh one nitpick, looks like the merge on CHANGES.rst put a line in the wrong place.)

you might @ a few folks who have merged PRs in the last year for another pair of eyes.

@tony Would you care to take a look?

@jcushman @lucaswiman I don't use the deferred fields part of django-model-utils.
I breezed through the code and it LGTM. I think that's the best I can offer.
I don't want to offer more since I don't have a Django website using that part of django-model-utils to test on fully.

Thanks @tony and @jcushman! I'll merge this and fix pep8 in another PR.