Skip to content

Conditional Linear Gaussian models

Creative Commons LicenseaGrUMinteractive online version
import pyagrum as gum
import pyagrum.clg as gclg
import pyagrum.clg.notebook as gclgnb
import pyagrum.lib.bn_vs_bn as gcm
import pyagrum.lib.notebook as gnb

Suppose we want to build a CLG with these specifications A=N(5,1)A={\cal N}(5,1), B=N(4,3)B={\cal N}(4,3) and C=2.A+3.B+N(3,2)C=2.A+3.B+{\cal N}(3,2)

model = gclg.CLG()
model.add(gclg.GaussianVariable("A", 5, 1))
model.add(gclg.GaussianVariable("C", 3, 2))
model.add(gclg.GaussianVariable("B", 4, 3))
model.addArc("A", "C", 2)
model.addArc("B", "C", 3)
model
G A A μ=5.000 σ=1.000 C C μ=3.000 σ=2.000 A->C 2.00 B B μ=4.000 σ=3.000 B->C 3.00

We can create a Conditional Linear Gaussian Bayesian networ(CLG model) using a SEM-like syntax.

A = 4.5 [0.3] means that the mean of the distribution for Gaussian random variable A is 4.5 and ist standard deviation is 0.3.

B = 3 + 0.8F [0.3] means that the mean of the distribution for the Gaussian random variable B is 3 and the standard deviation is 0.3.

pyagrum.CLG.SEM is a set of static methods to manipulate this kind of SEM.

sem2 = """
A=4.5 [0.3] # comments are allowed
F=7 [0.5]
B=3 + 1.2F [0.3]
C=9 + 2A + 1.5B [0.6]
D=9 + C + F[0.7]
E=9 + D [0.9]
"""
model2 = gclg.SEM.toclg(sem2)
gnb.show(model2)

svg

One can of course build the SEM from a CLG using pyagrum.CLG.SEM.tosem :

gnb.flow.row(
model,
"<pre><div align='left'>" + gclg.SEM.tosem(model) + "</div></pre>",
captions=["the first CLG model", "the SEM from the CLG"],
)
B=4[3] A=5[1] C=3+2A+3B[2]

the SEM from the CLG

And this SEM allows of course input/output format for CLG

gclg.SEM.saveCLG(model2, "out/model2.sem")
print("=== file content ===")
with open("out/model2.sem") as file:
for line in file:
print(line, end="")
print("====================")
=== file content ===
F=7.0[0.5]
B=3.0+1.2F[0.3]
A=4.5[0.3]
C=9.0+2.0A+1.5B[0.6]
D=9.0+F+C[0.7]
E=9.0+D[0.9]
====================
model3 = gclg.SEM.loadCLG("out/model2.sem")
gnb.sideBySide(model2, model3, captions=["saved model", "loaded model"])
import pickle
with open("out/testCLG.pkl", "bw") as f:
pickle.dump(model3, f)
model3
G F F μ=7.000 σ=0.500 B B μ=3.000 σ=0.300 F->B 1.20 D D μ=9.000 σ=0.700 F->D 1.00 C C μ=9.000 σ=0.600 B->C 1.50 A A μ=4.500 σ=0.300 A->C 2.00 C->D 1.00 E E μ=9.000 σ=0.900 D->E 1.00
model.dag().sizeArcs()
2
with open("out/testCLG.pkl", "br") as f:
copyModel3 = pickle.load(f)
copyModel3
G F F μ=7.000 σ=0.500 B B μ=3.000 σ=0.300 F->B 1.20 D D μ=9.000 σ=0.700 F->D 1.00 C C μ=9.000 σ=0.600 B->C 1.50 A A μ=4.500 σ=0.300 A->C 2.00 C->D 1.00 E E μ=9.000 σ=0.900 D->E 1.00

Compute some posterior using difference exact inference

ie = gclg.CLGVariableElimination(model2)
ie.updateEvidence({"D": 3})
print(ie.posterior("A"))
print(ie.posterior("B"))
print(ie.posterior("C"))
print(ie.posterior("D"))
print(ie.posterior("E"))
print(ie.posterior("F"))
v = ie.posterior("E")
print(v)
print(f" - mean(E|D=3)={v.mu()}")
print(f" - stdev(E|D=3)={v.sigma()}")
A:1.9327650111193468[0.28353638852446156]
B:-2.5058561897702[0.41002992170553515]
C:3.9722757598220895[0.5657771474513671]
D:3[0]
E:12.0[0.9]
F:-2.9836916234247597[0.32358490464094586]
E:12.0[0.9]
- mean(E|D=3)=12.0
- stdev(E|D=3)=0.9
gnb.sideBySide(
model2,
gclgnb.getInference(model2, evs={"D": 3}, size="3!"),
gclgnb.getInference(model2, evs={"D": 3, "F": 1}),
captions=["The CLG", "First inference", "Second inference"],
)

Approximated inference : MonteCarlo Sampling

Section titled “Approximated inference : MonteCarlo Sampling”

When the model is too complex for exact infernece, we can use forward sampling to generate 5000 samples from the original CLG model.

fs = gclg.ForwardSampling(model2)
fs.makeSample(5000).tocsv("./out/model2.csv")

We will use the generated database to do learning. But before, we can also compute posterior but without evidence :

ie = gclg.CLGVariableElimination(model2)
print("| 'Exact' inference | Results from sampling |")
print("|------------------------------------------|------------------------------------------|")
for i in model2.names():
print(f"| {ie.posterior(i)!s:40} | {gclg.GaussianVariable(i, fs.mean_sample(i), fs.stddev_sample(i))!s:40} |")
| 'Exact' inference | Results from sampling |
|------------------------------------------|------------------------------------------|
| A:4.499999999999998[0.3] | A:4.49775454042852[0.2995341686826186] |
| F:7.000000000000008[0.5000000000000002] | F:7.00532083686043[0.5031611132650285] |
| B:11.399999999999999[0.6708203932499367] | B:11.403212550758653[0.6781293275986302] |
| C:35.099999999999994[1.3162446581088183] | C:35.10737325782851[1.3259794458046597] |
| D:51.10000000000002[1.8364367672206963] | D:51.10516180911396[1.8462258911136797] |
| E:60.100000000000016[2.0451161336217565] | E:60.10787237189177[2.0538504556572508] |

Now with the generated database and the original model, we can calculate the log-likelihood of the model.

print("log-likelihood w.r.t orignal model : ", model2.logLikelihood("./out/model2.csv"))
log-likelihood w.r.t orignal model : -22044.92903269877

Use the generated database to do our RAvel Learning. This part needs some time to run.

## RAveL learning
learner = gclg.CLGLearner("./out/model2.csv")

We can get the learned_clg model with function learn_clg() which contains structure learning and parameter estimation.

learned_clg = learner.learnCLG()
gnb.sideBySide(model2, learned_clg, captions=["original CLG", "learned CLG"])

Compare the learned model’s structure with that of the original model’.

cmp = gcm.GraphicalBNComparator(model2.asDiscreteBN(), learned_clg.asDiscreteBN())
print(f"F-score(original_clg,learned_clg) : {cmp.scores()['fscore']}")
F-score(original_clg,learned_clg) : 0.8

Get the learned model’s parameters and compare them with the original model’s parameters using the SEM syntax.

gnb.flow.row(
"<pre><div align='left'>" + gclg.SEM.tosem(model2) + "</div></pre>",
"<pre><div align='left'>" + gclg.SEM.tosem(learned_clg) + "</div></pre>",
captions=["original sem", "learned sem"],
)
F=7.0[0.5] B=3.0+1.2F[0.3] A=4.5[0.3] C=9.0+2.0A+1.5B[0.6] D=9.0+F+C[0.7] E=9.0+D[0.9]

original sem
E=60.10787237189177[2.0538504556572508] F=7.00532083686043[0.5031611132650285] B=2.898429619177355+1.2140461699956744F[0.2944629697345253] D=6.003616108245801+1.055523634115265F+0.6273265465079879E[0.7159348138365407] A=4.49775454042852[0.2995341686826186] C=2.0055291843128202+1.1669186146783816A+0.6342778459853293B+0.40349203000370754D[0.45954471866868557]

learned sem

We can algo do parameter estimation only with function fitParameters() if we already have the structure of the model.

## We can copy the original CLG
copy_original = gclg.CLG(model2)
## RAveL learning again
RAveL_l = gclg.CLGLearner("./out/model2.csv")
## Fit the parameters of the copy clg
RAveL_l.fitParameters(copy_original)
copy_original
G A A μ=4.498 σ=0.300 C C μ=9.064 σ=0.595 A->C 1.96 F F μ=7.005 σ=0.503 B B μ=2.898 σ=0.294 F->B 1.21 D D μ=9.090 σ=0.691 F->D 0.99 B->C 1.51 C->D 1.00 E E μ=9.033 σ=0.902 D->E 1.00

We first create two CLG from two SEMs.

## TWO DIFFERENT CLGs
## FIRST CLG
clg1 = gclg.SEM.toclg("""
## hyper parameters
A=4[1]
B=3[5]
C=-2[5]
#equations
D=A[.2] # D is a noisy version of A
E=1+D+2B [2]
F=E+C+B+E [0.001]
""")
## SECOND CLG
clg2 = gclg.SEM.toclg("""
## hyper parameters
A=4[1]
B=3+A[5]
C=-2+2B+A[5]
#equations
D=A[.2] # D is a noisy version of A
E=1+D+2B [2]
F=E+C [0.001]
""")

This cell shows how to have a quick view of the differences

gnb.flow.row(clg1, clg2, gcm.graphDiff(clg1, clg2), gcm.graphDiffLegend(), gcm.graphDiff(clg2, clg1))
G A A B B A->B C C A->C D D A->D B->C E E B->E F F B->F C->F D->E E->F
G a->b overflow c->d Missing e->f reversed g->h Correct
G A A B B A->B C C A->C D D A->D B->C E E B->E F F B->F C->F D->E E->F

We compare the CLG models.

## We use the F-score to compare the two CLGs
cmp = gcm.GraphicalBNComparator(clg1.asDiscreteBN(), clg1.asDiscreteBN())
print(f"F-score(clg1,clg1) : {cmp.scores()['fscore']}")
cmp = gcm.GraphicalBNComparator(clg1.asDiscreteBN(), clg2.asDiscreteBN())
print(f"F-score(clg1,clg2) : {cmp.scores()['fscore']}")
F-score(clg1,clg1) : 1.0
F-score(clg1,clg2) : 0.7142857142857143
## The complete list of structural scores is :
print("score(clg1,clg2) :")
for score, val in cmp.scores().items():
print(f" - {score} : {val}")
score(clg1,clg2) :
- count : {'tp': 5, 'tn': 6, 'fp': 3, 'fn': 1}
- recall : 0.8333333333333334
- precision : 0.625
- fscore : 0.7142857142857143
- dist2opt : 0.41036907507483766
- sid : 3
## We create a simple CLG with 3 variables
clg = gclg.CLG()
## prog=« sigma=2;X=N(5);Y=N(3);Z=X+Y »
A = gclg.GaussianVariable(mu=2, sigma=1, name="A")
B = gclg.GaussianVariable(mu=1, sigma=2, name="B")
C = gclg.GaussianVariable(mu=2, sigma=3, name="C")
idA = clg.add(A)
idB = clg.add(B)
idC = clg.add(C)
clg.addArc(idA, idB, 1.5)
clg.addArc(idB, idC, 0.75)
## We can show it as a graph
original_clg = gclgnb.CLG2dot(clg)
original_clg
G A A μ=2.000 σ=1.000 B B μ=1.000 σ=2.000 A->B 1.50 C C μ=2.000 σ=3.000 B->C 0.75
fs = gclg.ForwardSampling(clg)
fs.makeSample(10)

<pyagrum.clg.forwardSampling.ForwardSampling at 0x1139d3d90>

print("A's sample_variance: ", fs.variance_sample(0))
print("B's sample_variance: ", fs.variance_sample("B"))
print("C's sample_variance: ", fs.variance_sample(2))
A's sample_variance: 0.5807336622371126
B's sample_variance: 5.935408099966504
C's sample_variance: 16.53043217909329
print("A's sample_mean: ", fs.mean_sample("A"))
print("B's sample_mean: ", fs.mean_sample("B"))
print("C's sample_mean: ", fs.mean_sample("C"))
A's sample_mean: 2.5917571791071863
B's sample_mean: 4.957140355489152
C's sample_mean: 7.443758132884577
fs.toarray()
array([[ 2.78142432, 3.09628358, 5.92146504],
[ 2.70695407, 6.13473714, 6.87705979],
[ 3.55305019, 5.94231227, 12.62763698],
[ 1.87199497, 6.44577512, 9.01798136],
[ 1.7682016 , 2.40170462, 10.47649221],
[ 2.10449903, 1.31127139, 5.70704977],
[ 2.64301526, 8.66337213, 8.08904898],
[ 2.82601545, 4.82366932, 4.51027423],
[ 4.10166276, 8.32488968, 12.8619973 ],
[ 1.56075413, 2.4273883 , -1.65142434]])
## export to dataframe
fs.topandas()
A B C
0 2.781424 3.096284 5.921465
1 2.706954 6.134737 6.877060
2 3.553050 5.942312 12.627637
3 1.871995 6.445775 9.017981
4 1.768202 2.401705 10.476492
5 2.104499 1.311271 5.707050
6 2.643015 8.663372 8.089049
7 2.826015 4.823669 4.510274
8 4.101663 8.324890 12.861997
9 1.560754 2.427388 -1.651424
## export to csv
fs.makeSample(10000)
fs.tocsv("./out/samples.csv")

The module allows to investigale more deeply into the learning algorithm.

We first create a random CLG model with 5 variables.

## Create a new random CLG
clg = gclg.randomCLG(nb_variables=5, names="ABCDE")
## Display the CLG
print(clg)
A=-4.871477831690884[5.388075148539592]
B=2.197845351657622+7.323819404786844A[4.788753303041101]
C=0.4981449389243364[7.335642185659528]
D=3.9551259054076464-5.850050097939021B-2.6744509887944083C+1.2860926187052453A[9.365361910728655]
E=-3.9163636059860387+1.8547773180869067B-4.96076392395403D[1.7009106894150645]

We then do the Forward Sampling and CLGLearner.

n = 20 # n is the selected values of MC number n in n-MCERA
K = 10000 # K is the list of selected values of number of samples
Delta = 0.05 # Delta is the FWER we want to control
## Sample generation
fs = gclg.ForwardSampling(clg)
fs.makeSample(K).tocsv("./out/clg.csv")
## Learning
RAveL_l = gclg.CLGLearner("./out/clg.csv", n_sample=n, fwer_delta=Delta)

We use the PC algorithme to learn the structure of the model.

## Use the PC algorithm to get the skeleton
C = RAveL_l.PC_algorithm(order=clg.nodes(), verbose=False)
print("The final skeleton is:\n", C)
The final skeleton is:
{0: {1, 2}, 1: set(), 2: {1}, 3: set(), 4: {0}}
## Create a Mixedgraph to display the skeleton
RAveL_MixGraph = gum.MixedGraph()
## Add variables
for i in range(len(clg.names())):
RAveL_MixGraph.addNodeWithId(i)
RAveL_MixGraph.setName(i, clg.variable(i).name())
## Add arcs and edges
for father, kids in C.items():
for kid in kids:
if father in C[kid]:
RAveL_MixGraph.addEdge(father, kid)
else:
RAveL_MixGraph.addArc(father, kid)
RAveL_MixGraph
0 (0) B 1 (1) E 0->1 2 (2) D 0->2 2->1 3 (3) C 4 (4) A 4->0
## Create a BN with the same structure as the CLG
bn = clg.asDiscreteBN()
## Compare the result above with the EssentialGraph
Real_EssentialGraph = gum.EssentialGraph(bn)
Real_EssentialGraph
0 B 1 E 0->1 2 D 0->2 4 A 0->4 2->1 3 C 3->2 4->2
## create a CLG from the skeleton of PC algorithm
clg_PC = gclg.CLG()
for node in clg.nodes():
clg_PC.add(clg.variable(node))
for father, kids in C.items():
for kid in kids:
clg_PC.addArc(father, kid)
## Compare the structure of the created CLG and the original CLG
print(f"F-score : {clg.structuralFScore(clg_PC)}")
F-score : 0.28571428571428575

We can also do the parameter learning.

id2mu, id2sigma, arc2coef = RAveL_l.estimate_parameters(C)
for node in clg.nodes():
print(f"Real Value: node {node} : mu = {clg.variable(node)._mu}, sigma = {clg.variable(node)._sigma}")
print(f"Estimation: node {node} : mu = {id2mu[node]}, sigma = {id2sigma[node]}")
for arc in clg.arcs():
print(f"Real Value: arc {arc} : coef = {clg.coefArc(*arc)}")
print(f"Estimation: arc {arc} : coef = {(arc2coef[arc] if arc in arc2coef else '-')}")
Real Value: node 0 : mu = 2.197845351657622, sigma = 4.788753303041101
Estimation: node 0 : mu = 2.1963609633449366, sigma = 4.769150032317679
Real Value: node 1 : mu = -3.9163636059860387, sigma = 1.7009106894150645
Estimation: node 1 : mu = -3.899988950331931, sigma = 1.7010089552805656
Real Value: node 2 : mu = 3.9551259054076464, sigma = 9.365361910728655
Estimation: node 2 : mu = 2.22978675911412, sigma = 21.9216450057524
Real Value: node 3 : mu = 0.4981449389243364, sigma = 7.335642185659528
Estimation: node 3 : mu = 0.5951400523850416, sigma = 7.426821747344648
Real Value: node 4 : mu = -4.871477831690884, sigma = 5.388075148539592
Estimation: node 4 : mu = -4.896279533742537, sigma = 5.450120211557237
Real Value: arc (0, 1) : coef = 1.8547773180869067
Estimation: arc (0, 1) : coef = 1.8585314047050532
Real Value: arc (0, 2) : coef = -5.850050097939021
Estimation: arc (0, 2) : coef = -5.669473421944598
Real Value: arc (4, 0) : coef = 7.323819404786844
Estimation: arc (4, 0) : coef = 7.328506223375432
Real Value: arc (2, 1) : coef = -4.96076392395403
Estimation: arc (2, 1) : coef = -4.960137385952034
Real Value: arc (3, 2) : coef = -2.6744509887944083
Estimation: arc (3, 2) : coef = -
Real Value: arc (4, 2) : coef = 1.2860926187052453
Estimation: arc (4, 2) : coef = -