In this paper, we study linear convolutional networks with one-dimensional filters and arbitrary strides. The neuromanifold of such a network is a semialgebraic set, represented by a space of polynomials admitting specific factorizations. Introducing a recursive algorithm, we generate polynomial equations whose common zero locus corresponds to the Zariski closure of the corresponding neuromanifold. Furthermore, we explore the algebraic complexity of training these networks employing tools from metric algebraic geometry. Our findings reveal that the number of all complex critical points in the optimization of such a network is equal to the generic Euclidean distance degree of a Segre variety. Notably, this count significantly surpasses the number of critical points encountered in the training of a fully connected linear network with the same number of parameters.

The set of functions parameterized by a linear fully connected neural network is a determinantal variety. We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. Examples of such group actions are translations or 90 degree rotations on images. We describe such equivariant or invariant subvarieties as direct products of determinantal varieties, from which we deduce their dimension, degree, Euclidean distance degree, and their singularities. We fully characterize invariance for arbitrary permutation groups, and equivariance for cyclic groups. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks in terms of sparsity and weight-sharing properties. We prove that all invariant linear functions can be parameterized by a single linear autoencoder with a weight-sharing property imposed by the cycle decomposition of the considered permutation. The space of rank-bounded equivariant functions has several irreducible components, so it cannot be parameterized by a single network—but each irreducible component can. Finally, we show that minimizing the squared-error loss on our invariant or equivariant networks reduces to minimizing the Euclidean distance from determinantal varieties via the Eckart–Young theorem.

Neural networks are central to modern machine learning, with applications that range from computer vision to natural language processing. Despite their success, their mathematical foundations are barely understood. At the heart of every such model lies a training procedure that amounts to solving a highly nonconvex optimization problem with many potential solutions, yet optimization algorithms often find parameters that not only fit the data but also generalize well to unseen samples. Why neural networks exhibit this favorable behavior, and how architectural choices influence it, remain fundamental open questions that call for new mathematical tools.

This thesis suggests one promising approach, neuroalgebraic geometry, whose research program is to study neural networks through the lens of algebraic geometry. In this framework, nonlinearities such as activation functions are replaced with algebraic counterparts, for instance polynomials, so that the resulting models become amenable to rigorous algebro-geometric analysis. Since polynomials are universal approximators, by a limit argument, the methods developed in neuroalgebraic geometry can extend beyond the polynomial realm and thus bridge the gap between algebraic models and practical neural networks.

Through neuroalgebraic geometry, we study the function space parameterized by a given neural network architecture, which we refer to as the neuromanifold. Its dimension and volume reflect how rich the model is and how well it can generalize from data. Singular points, places where the neuromanifold is not regular, characterize implicit biases that arise during training. The analysis of the parametrization map relates to the identifiability of neural networks, a property that is essential for the design of equivariant architectures in which symmetries of the data are encoded into the model. Finally, viewing optimization through this geometric lens connects the landscape of the loss function to the underlying structure of the neuromanifold. The algebraic setting makes these analyses more feasible since it makes the ambient space of the neuromanifold finite-dimensional.

The main goal of this thesis is to analyze the functionality of two important architectures, multilayer perceptrons (MLPs) and convolutional neural networks (CNNs), through the lens of algebraic geometry.

In Paper A, we present a position paper that introduces and motivates the emerging research area of neuroalgebraic geometry. We construct a dictionary between algebro-geometric concepts (such as dimension, degree, and singularities) and key machine learning phenomena (including sample complexity, expressivity, and implicit bias). Along the way, the paper provides a concise literature overview and argues for new connections at the intersection of algebraic geometry and machine learning.

In Paper B, we investigate linear convolutional networks with single-channel and one-dimensional filters. By examining their neuromanifold, we provide its dimension and singularities. Furthermore, by considering optimization with squared loss, we show that the critical points of the parameterization corresponding to spurious points are not attracted by gradient-based optimization once all strides are larger than one.

In Paper C, which continues the investigation from Paper B, we introduce a recursive algorithm that generates the polynomial equations defining the Zariski closure of the neuromanifold of linear convolutional networks. We further provide the exact number of (complex) critical points that arise when training these networks with squared loss and generic data.

In Paper D, we examine linear invariant and equivariant networks under permutation groups. We determine the dimension, degree, and singular locus of the neuromanifold for these models. We then analyze the number of (complex) critical points that can arise during training. Furthermore, we show that the neuromanifold of linear equivariant networks comprises many irreducible components that cannot be parameterized by a single fixed architecture, and thus the choice of architecture determines which irreducible component we parameterize.

In Paper E, which is our first exploration of non-linear activation functions, we analyze single-channel, one-dimensional-filter convolutional networks with monomial activation functions. We show that its neuromanifold, once projectified, is birational to a Segre--Veronese variety, a well-known object in classical algebraic geometry. We then describe its algebraic invariants such as dimension and degree and characterize its singular points, including their types. Finally, we provide an exact formula for the number of (complex) critical points arising when training with generic data under squared loss optimization.

In Paper F, we investigate both MLPs and CNNs with generic polynomial activation functions. We prove that no continuous symmetry exists in either model, i.e., the generic fiber of the parameterization is finite. Consequently, the dimension of the neuromanifolds coincides with the number of parameters. Furthermore, we show that in both models, certain subnetworks correspond to singular points of the neuromanifold. Finally, for CNNs the parameters associated with these subnetworks are not critically exposed, whereas for MLPs they are critically exposed, meaning that they appear as critical points of the loss function with nonzero probability over the data distribution.

Although the main thrust of this thesis concerns the geometry and optimization of neural networks, the perspective developed here---framing learning problems as algebraic optimization over structured sets---also motivated a new approach to a classical problem in signal processing. 

In Paper G, we address the problem of multireference alignment (MRA), where multiple noisy observations of a one-dimensional signal are available, each subjected to an unknown circular shift. The objective is to reconstruct the underlying signal up to circular shift. We introduce a novel algorithm that minimizes a distance function defined on the manifold of signals whose second-order moments agree with those estimated from the observations. We analyze the optimization problem both in the finite- and infinite-data regimes. In the latter, we show that the true signal is always a critical point of the loss function, and our empirical results show that it corresponds to a global minimum.

Neuronnät är centrala för modern maskininlärning, med tillämpningar som sträcker sig från datorseende till naturlig språkbearbetning. Trots deras framgång är de matematiska grunderna inte väletablerade. I kärnan av varje sådan modell finns en träningsprocedur som syftar till att lösa ett icke-konvext optimeringsproblem med många potentiella lösningar, ändå hittar optimeringsalgoritmer ofta parametrar som inte bara anpassar sig till data utan också generaliserar väl till osedda exempel. Varför neuronnät uppvisar detta gynnsamma beteende, och hur arkitektoniska val påverkar det, är fortfarande grundläggande öppna frågor som kräver nya matematiska verktyg.

Denna avhandling föreslår en lovande ansats, neuroalgebraisk geometri, vars forskningsprogram är att studera neuronnät genom algebraisk geometri. I detta ramverk ersätts icke-linjäriteter såsom aktiveringsfunktioner med algebraiska motsvarigheter, till exempel polynom, så att de resulterande modellerna blir mer lämpade för rigorös algebro-geometrisk analys. Eftersom polynom är universella approximatorer kan metoderna som utvecklas inom neuroalgebraisk geometri (genom att ta de relevanta gränsvärdena) sträcka sig bortom polynomvärlden och därmed överbrygga klyftan mellan algebraiska modeller och praktiska neuronnät.

Genom neuroalgebraisk geometri studerar vi funktionsrummet som parameteriseras av en given neuronnätsarkitektur, vilket vi kallar neuromångfald. Dess dimension och volym återspeglar hur rik modellen är och hur väl den kan generalisera från data. Singulära punkter, platser där neuromångfalden inte är reguljär, karakteriserar implicita bias som uppstår under träning. Analysen av parameteriseringsavbildningen relaterar till neuronnätets identifierbarhet, en egenskap som är avgörande för utformningen av ekvivarianta arkitekturer där datasymmetrier kodas in i modellen. Att betrakta optimering ur detta geometriska perspektiv kopplar förlustfunktionens landskap till den underliggande strukturen hos neuromångfalden. Den algebraiska miljön gör dessa analyser mer genomförbara eftersom det omgivande rummet för neuromångfalden blir ändligtdimensionellt.

Huvudmålet med denna avhandling är att analysera funktionaliteten hos två viktiga arkitekturer, flerlagersperceptroner (MLP:er) och faltningsnätverk (CNN:er), genom algebraisk geometri.

I Artikel A presenterar vi ett ståndpunktsdokument som introducerar och motiverar det framväxande forskningsområdet neuroalgebraisk geometri. Vi konstruerar en ordbok mellan algebro-geometriska begrepp (såsom dimension, grad och singulariteter) och centrala fenomen i maskininlärning (inklusive samplingskomplexitet, expressivitet och implicit bias). Samtidigt ger artikeln en koncis litteraturöversikt och argumenterar för nya kopplingar i skärningspunkten mellan algebraisk geometri och maskininlärning.

I Artikel B undersöker vi linjära faltningsnät med en kanal och endimensionella filter. Genom att studera deras neuromångfald anger vi dess dimension och singulariteter. Genom att betrakta optimering med kvadratisk förlust, visar vi vidare att de kritiska punkterna för parameteriseringen som motsvarar spuriösa punkter inte attraheras av gradientbaserad optimering när alla stegstorlekar (eng. strides) är större än ett.

I Artikel C, som fortsätter undersökningen från Artikel B, introducerar vi en rekursiv algoritm som genererar de polynomekvationer som definierar Zariski-slutningen av neuromångfalden för linjära faltningsnät. Vi ger vidare det exakta antalet (komplexa) kritiska punkter som uppstår när dessa nät tränas med kvadratisk förlust och generiska data.

I Artikel D undersöker vi linjära invarianta och ekvivarianta nätverk under permutationsgrupper. Vi bestämmer dimension, grad och singulärmängd hos neuromångfalden för dessa modeller. Därefter analyserar vi antalet (komplexa) kritiska punkter som kan uppstå under träning. Vidare visar vi att neuromångfalden för linjära ekvivarianta nätverk består av många irreducibla komponenter som inte kan parameteriseras av en enda fix arkitektur, och därmed avgör arkitekturvalet vilken irreducibel komponent vi parameteriserar.

I Artikel E, som är vår första utforskning av icke-linjära aktiveringsfunktioner, analyserar vi faltningsnät med en kanal och endimensionella filter med monom som aktiveringsfunktioner. Vi visar att dess neuromångfald, när den projektiviseras, är birationell med en Segre--Veronese-varietet, ett välkänt objekt inom klassisk algebraisk geometri. Vi beskriver därefter dess algebraiska invarianter såsom dimension och grad och karakteriserar dess singulära punkter, inklusive deras typer. Slutligen ger vi en exakt formel för antalet (komplexa) kritiska punkter som uppstår vid träning med generiska data under optimering med kvadratisk förlust.

I Artikel F undersöker vi både MLP:er och CNN:er med generiska polynom som aktiveringsfunktioner. Vi bevisar att inga kontinuerliga symmetrier finns i någon av modellerna, det vill säga den generiska fibern av parameteriseringen är ändlig. Följaktligen sammanfaller neuromångfaldernas dimension med antalet parametrar. Vidare visar vi att det i båda modellerna finns vissa delnätverk som motsvarar singulära punkter i neuromångfalden. Slutligen är parametrarna associerade med dessa delnätverk, för CNN:er, inte kritiskt exponerade, medan de för MLP:er är kritiskt exponerade, vilket innebär att de uppträder som kritiska punkter för förlustfunktionen med nollskild sannolikhet över datafördelningen.

Även om huvudinriktningen i denna avhandling rör geometri och optimering av neuronnät har det perspektiv som utvecklas här, där inlärningsproblem ramas in som algebraisk optimering över strukturerade mängder, också motiverat en ny ansats till ett klassiskt problem inom signalbehandling.

I Artikel G behandlar vi flerreferensorientering (MRA), ett problem där flera brusiga observationer av en endimensionell signal finns tillgängliga, var och en utsatt för en okänd cirkulär förskjutning. Målet är att rekonstruera den underliggande signalen upp till cirkulär förskjutning. Vi introducerar en ny algoritm som minimerar en avståndsfunktion definierad på mångfalden av signaler vars andra ordningens moment överensstämmer med dem som skattas från observationerna. Vi analyserar optimeringsproblemet både för ändlig och oändlig datamängd. I det senare visar vi att den sanna signalen alltid är en kritisk punkt för förlustfunktionen, och våra empiriska resultat visar att den motsvarar ett globalt minimum.

We study convolutional neural networks with monomial activation functions. Specifically, we prove that their parameterization map is regular and is an isomorphism almost everywhere, up to rescaling the filters. By leveraging on tools from algebraic geometry, we explore the geometric properties of the image in function space of this map - typically referred to as neuromanifold. In particular, we compute the dimension and the degree of the neuromanifold, which measure the expressivity of the model, and describe its singularities. Moreover, for a generic large dataset, we derive an explicit formula that quantifies the number of critical points arising in the optimization of a regression loss.

We study the geometry of linear networks with one-dimensional convolutional layers. The function spaces of these networks can be identified with semialgebraic families of polynomials admitting sparse factorizations. We analyze the impact of the network’s architecture on the function space’s dimension, boundary, and singular points. We also describe the critical points of the network’s parameterization map. Furthermore, we study the optimization problem of training a network with the squared error loss. We prove that for architectures where all strides are larger than one and generic data, the nonzero critical points of that optimization problem are smooth interior points of the function space. This property is known to be false for dense linear networks and linear convolutional networks with stride one.

Let G be a Frobenius group. Using character theory, it is proved that the Frobenius kernel of G is a normal subgroup of G, which is well-known as a Frobenius theorem. There is no known character-free proof for Frobenius theorem. In this note, we prove it, by assuming that Frobenius groups are non-simple. Also, we prove that whether K is a subgroup of G or not, Sylow 2-subgroups of G are either cyclic or generalized quaternion group. Also by assuming some additional arithmetical hypothesis on G we prove Frobenius theorem. We should mention that our proof is character-free.

Multireference alignment (MRA) refers to the problem of recovering a signal from noisy samples subject to random circular shifts. Expectation--maximization (EM) and variational approaches use statistical modeling to achieve high accuracy at the cost of solving computationally expensive optimization problems. The method of moments, instead, achieves fast reconstructions by utilizing the power spectrum and bispectrum to determine the signal up to shift. Our approach combines the two philosophies by viewing the power spectrum as a manifold on which to constrain the signal. We then maximize the data likelihood function on this manifold with a gradient-based approach to estimate the true signal. Algorithmically, our method involves iterating between template alignment and projections onto the manifold. The method offers increased speed compared to EM and demonstrates improved accuracy over bispectrum-based methods.