Welcome to ExCore
ExCore is a Configuration/Registry System designed for deeplearning, with some utils.
What inspired ExCore
Configuration System
The configuration and registry system is a great design. One of it's biggest challenges is that there is a gap between plain text config files(e.g. yaml and json) and python. The loudest complains are about editing features -- config auto-completion and code navigation. It's a awful experience when your program break down just because the name in config is misspelled.
The second challenge is that the codebase is becoming more and more complicated and nesting. Learner must pay more efforts on it. But this is not the topic here, just to mention that to solve this I am starting another project -- CodeSlim(Just started).
Back to the config system itself, since community has been suffering from the plain text config files for a long time, some are turn to using Python as config. MMEngine introduced a pure Python style configuration just for class code navigation.
from torch.optim import SGD
optimizer = dict(type=SGD, lr=0.1)
detectron2 has a more elegant implementation -- LazyCall, which discards the type and is able to instantiate recursively.
from detectron2.config import LazyCall
from torch.optim import SGD
optimizer = LazyCall(SGD)(lr=0.1)
However, both of them only provide auto-completion and navigation for the class instead of their arguments. This is one of the reasons why ExCore was created.
Registry System
The blog of ppwwyyxx inspired me a lot. To reduce the unnecessary imports, ExCore provides LazyRegistry, which store the mappings of class/function name to its qualname and dump the mappings to local. When config parsing, the necessary modules will be imported.
Rather than calling it a registry, it's more like providing a tagging feature. With the feature, ExCore can find all class/function and statically analysis them, then dump the results in local to support some editing features to config files, see config extension.
Plain Text Configuration
To be honest, I don't like the Python style config at all. The readability of Python is not as good as YAML or TOML. If using Python, why not just instantiate all modules in a file, then import it if need.
# a.py
model = xxx(a=1,b=2)
optimizer = xx(model.parameters())
...
# usage
from configs import a
model = a.model
optimizer = a.optimizer
...
ExCore also supports python style config, see here.
Some defects
- Too static.
- Not fully tested.
Others
ExCore is just a toy now. Dumping data into local is a rough workaround. Build an LSP server may be a more elegant way.