So far I've found the most meaningful and reusable representations can be formatted through a gated geometric hierarchy. I'm currently running roughly 50k images through the VAE in order to assess the capacity of the model's components before refactor or reassessment. So far the results are promising for synthetic supervised local patch geometric contribution bias being a very real potential. The model learns to predict the classification elements and then the model no longer requires the transformer blocks, so the gates can be snapped off and the model turned into a fragment of it's larger self. A form of hardened crystalline.

The gates are nearly deterministic between trains, however the classification elements are non-determinant - which means the model is learning to bias in specific routes beyond the current stage in order to justify classification goals. The gates themselves are producing utilizable feature information however, so the outcomes are promising on the refactor.

So far the patch features are showing the most robust reusability potential, but that's only about 120 images or so total, the 50k 15 category test will be the real measure.

Surprisingly the gate statistics are essentially useless, nearly identical through all stages.

50k test completed using synthetic data extracted from flux for another project;
https://huggingface.co/datasets/AbstractPhil/synthetic-characters

This is more than enough inference information to get a fair measure as to which features are the most helpful and which aren't so useful.

The results are here as well as the runner;
https://huggingface.co/AbstractPhil/grid-geometric-multishape/tree/main/50k_results

It requires the cell 1 model code and then it'll run.

So what we do here, is snap off the classifier and utilize the various features in cosine similarity conjunction. The accuracy of the tested model is roughly 93% 3-4 shape shared space in the patches, so this can be greatly expanded but it requires additional computational power.

The 3-4 shape shared space should be more than enough pretraining for this hypothesis; which seems to be building more and more potency as something beyond a possibility. This is most definitely a measurable phenomena. Geometric structure most definitely can be analyzed and compacted into useful discriminative features in order to apply a learned bias. How USEFUL those features are? Well, they're pretty discriminative, so there needs to be more tests.

This leaves many questions. Predominantly, the singular one that will be required; can the patches be made smaller if the mathematics are condensed and the shared attention is expanded, and how many patches can this actually support within a nearly-instant computation window?

Does this require the geometric transformers to train or can it learn useful features independently?

Can this benefit from captured embeds in differential conjunction sharing space with a powerful text encoder such as Qwen 2.5 instruct?

Will these patches actually provide attention use down the chain to a diffusion model, or will the mechanism simply get lost in the noise?