STI correlation table : A small data science project for me and a Christmas present for you.

The past few weeks after the bar exams was a balancing act. The wife and kids are out of the country and I was supposed to have 2-3 weeks of intense partying. But I am too old to party anymore and can't seem to find any karaoke kakis no matter how hard I tried. Fortunately, I had quite a few intense D&D sessions.

While I should avoid doing useful and practical stuff, one of my bigger fears is that I would lose some of my ability to code once I begin legal practice, so I tried to do a very small project just to maintain some currency in statistical programming. 

I used R Studio to create my program. Specifically, I taught myself the 'tseries' package function get.hist.quote() allows me to download 10 years of daily price information from Yahoo finance into a "zoo" data structure. I created a new vector containing the lagged difference of logarithmic price information extracted and tried to find the correlation between each pair of stocks in the STI index. The final result is arranged in a matrix that I eventually transferred and beautified using Excel.

If done badly, you just end up with a massive data dump like the diagram below, but one piece of information is interesting. 

The two stocks in the STI which are most negatively correlated with each other is Ascendas REIT and HPH Trust. So if you want to build a portfolio that gives acceptable yields but one component would move north when the other moves south, think no further than considering this pair of stocks.

Eventually, I will reuse this piece of code to loop through the REITs in my margin portfolio - the twin stocks which are poorest correlated to each other would be accumulated first when my legal career begins to start paying the bills.

There are a lot of problems if like me, you wish to become an arm-chair data scientist. Local data sources can be quite dirty and effort is required to cleanse the data before performing statistical operations on it. The folks in US have this database called CRSP so they don't have the same problems that we do.

As we become a Smart City, we need to collectively lobby for better data to assist our home-brewing research efforts. 

A good place to start is to push SGX to publish daily prices fro 10 years or more in CSV format and allow us to download for free.

-->

	NS8U.SI	MC0.SI	E5H.SI	S58.SI	Y92.SI	BS6.SI	U11.SI	BN4.SI	C6L.SI	J37.SI	S63.SI	C07.SI	O39.SI	Z74.SI	U96.SI	A17U.SI	H78.SI	C52.SI	C31.SI	D05.SI	F34.SI	J36.SI	G13.SI	T39.SI	C09.SI	S68.SI	C38U.SI	C61U.SI	U14.SI	CC3.SI
NS8U.SI	1	-0.01649074	0.08006526	0.03437739	-0.02271943	0.001976555	-0.01714741	0.0291452	-0.03804963	-0.00181447	-0.002143713	-0.03373092	-0.01745041	0.01350265	-0.01873741	-0.07888264	0.02561792	0.001747832	-0.01439667	-0.0239544	0.02870607	0.03353878	-0.01990264	-0.02103148	-0.03980885	-0.02585094	-0.06884904	-0.03457492	-0.01365725	0.03049565
MC0.SI	-0.016490741	1	0.22428522	0.11982527	0.06870477	0.289172417	0.38535539	0.38529998	0.24924094	0.14890437	0.301627967	0.28407001	0.38562593	0.28011956	0.37624669	0.23206764	0.28769031	0.220561226	0.38513376	0.40098622	0.25809527	0.1834527	0.31098893	0.25905568	0.33281166	0.36322738	0.27043672	0.30283243	0.3261733	0.15030383
E5H.SI	0.080065262	0.22428522	1	0.22769091	0.03699381	0.36945508	0.39693446	0.44637492	0.31072605	0.2395066	0.326634858	0.36693775	0.4289631	0.29238151	0.40775923	0.34142641	0.28313224	0.259453257	0.37599665	0.41057872	0.42843307	0.21457463	0.36905984	0.26144486	0.37502287	0.43454705	0.29909444	0.24476941	0.35168081	0.15619732
S58.SI	0.034377386	0.11982527	0.22769091	1	0.05790154	0.216021458	0.25354003	0.25599969	0.23572239	0.20900307	0.18420917	0.20977059	0.26932947	0.15689313	0.21752777	0.19612476	0.19539789	0.195891535	0.23277465	0.2596309	0.25771458	0.14403482	0.20691614	0.20811046	0.23903856	0.24499734	0.17341389	0.17039896	0.25570784	0.07791024
Y92.SI	-0.022719426	0.06870477	0.03699381	0.05790154	1	0.064567021	0.07764786	0.04750671	0.05353912	0.07619748	0.082530531	0.10093112	0.08891431	0.08087801	0.05196155	0.06217726	0.08520861	0.107918898	0.04845807	0.07637871	0.04328505	0.05911279	0.06702228	0.07761691	0.05065339	0.07917218	0.06962906	0.07473872	0.10227041	0.02102888
BS6.SI	0.001976555	0.28917242	0.36945508	0.21602146	0.06456702	1	0.44027783	0.43875593	0.33846931	0.25214292	0.328315877	0.36698295	0.45752734	0.29602569	0.44025551	0.27938218	0.30299275	0.276739788	0.38868446	0.45722384	0.34782568	0.19111988	0.39814831	0.2886306	0.42622874	0.48528919	0.26183488	0.23073046	0.40895875	0.19855817
U11.SI	-0.017147411	0.38535539	0.39693446	0.25354003	0.07764786	0.440277832	1	0.57154863	0.48476742	0.27152083	0.457089246	0.47024825	0.72835147	0.48608922	0.5340714	0.42972647	0.3649143	0.386898657	0.51746853	0.73183903	0.4143472	0.21664361	0.39372695	0.38985211	0.58237862	0.5953527	0.43132723	0.32108939	0.52803248	0.22556079
BN4.SI	0.029145201	0.38529998	0.44637492	0.25599969	0.04750671	0.438755934	0.57154863	1	0.43856284	0.32552104	0.429121452	0.46362049	0.5994581	0.41034301	0.69070048	0.41683628	0.35455357	0.35730854	0.47605935	0.59590919	0.49064251	0.28494713	0.42091553	0.36954329	0.54365724	0.56360353	0.40683629	0.31047079	0.47130496	0.26631419
C6L.SI	-0.038049633	0.24924094	0.31072605	0.23572239	0.05353912	0.338469307	0.48476742	0.43856284	1	0.21600815	0.395958631	0.39295215	0.51424073	0.42017698	0.41501125	0.36033821	0.28348861	0.362767032	0.4137702	0.50311068	0.35646697	0.18469909	0.33062188	0.35288161	0.48955387	0.47666378	0.3618681	0.26820728	0.42431783	0.20026937
J37.SI	-0.00181447	0.14890437	0.2395066	0.20900307	0.07619748	0.252142924	0.27152083	0.32552104	0.21600815	1	0.227788727	0.37219226	0.29573806	0.17147439	0.29458861	0.25112644	0.26973546	0.184230899	0.23725225	0.29787573	0.27328861	0.6036294	0.18902306	0.15386893	0.28765028	0.28343214	0.28447106	0.19547268	0.27837935	0.18909735
S63.SI	-0.002143713	0.30162797	0.32663486	0.18420917	0.08253053	0.328315877	0.45708925	0.42912145	0.39595863	0.22778873	1	0.38860136	0.46651744	0.39163357	0.40452021	0.29131454	0.28157484	0.413562306	0.38588315	0.43843959	0.32028658	0.18559713	0.29554308	0.34405217	0.4198056	0.41190999	0.30214922	0.2841591	0.41146983	0.23840084
C07.SI	-0.033730921	0.28407001	0.36693775	0.20977059	0.10093112	0.36698295	0.47024825	0.46362049	0.39295215	0.37219226	0.388601359	1	0.49953857	0.39010407	0.4147492	0.37999075	0.32301672	0.397992396	0.39530379	0.47972136	0.37509965	0.31211286	0.31073963	0.34256087	0.46003652	0.47045363	0.39784013	0.31895202	0.43042803	0.24528196
O39.SI	-0.017450407	0.38562593	0.4289631	0.26932947	0.08891431	0.45752734	0.72835147	0.5994581	0.51424073	0.29573806	0.466517443	0.49953857	1	0.51644195	0.5555458	0.45753488	0.36930716	0.449979026	0.53625647	0.73723373	0.43271316	0.23977154	0.41478762	0.41837654	0.61554086	0.60466304	0.4509628	0.34313138	0.54606635	0.25593512
Z74.SI	0.01350265	0.28011956	0.29238151	0.15689313	0.08087801	0.296025693	0.48608922	0.41034301	0.42017698	0.17147439	0.391633568	0.39010407	0.51644195	1	0.39670609	0.34753836	0.2342488	0.408498882	0.37437458	0.48016237	0.33900772	0.13996073	0.31663354	0.36407193	0.40480005	0.40296097	0.34203842	0.29364786	0.39279021	0.27307262
U96.SI	-0.018737407	0.37624669	0.40775923	0.21752777	0.05196155	0.440255507	0.5340714	0.69070048	0.41501125	0.29458861	0.40452021	0.4147492	0.5555458	0.39670609	1	0.39106173	0.3324796	0.339221666	0.41455912	0.5477585	0.44026466	0.26556809	0.37526593	0.32937134	0.50096289	0.52259791	0.37455918	0.31229395	0.4605627	0.23054038
A17U.SI	-0.078882641	0.23206764	0.34142641	0.19612476	0.06217726	0.279382183	0.42972647	0.41683628	0.36033821	0.25112644	0.291314536	0.37999075	0.45753488	0.34753836	0.39106173	1	0.29121803	0.33286798	0.38369278	0.42954732	0.32730431	0.21065654	0.26825193	0.30966994	0.43852925	0.48568479	0.53005783	0.39737168	0.40434492	0.20958946
H78.SI	0.025617924	0.28769031	0.28313224	0.19539789	0.08520861	0.302992754	0.3649143	0.35455357	0.28348861	0.26973546	0.281574842	0.32301672	0.36930716	0.2342488	0.3324796	0.29121803	1	0.203622806	0.36632011	0.36066106	0.3066901	0.23771698	0.27643366	0.24498102	0.36459753	0.35997059	0.32469149	0.27110137	0.36181661	0.18855114
C52.SI	0.001747832	0.22056123	0.25945326	0.19589153	0.1079189	0.276739788	0.38689866	0.35730854	0.36276703	0.1842309	0.413562306	0.3979924	0.44997903	0.40849888	0.33922167	0.33286798	0.20362281	1	0.34257523	0.37824142	0.293783	0.15368549	0.302259	0.36128128	0.3426566	0.39085182	0.33703254	0.25433843	0.40350562	0.22582496
C31.SI	-0.014396668	0.38513376	0.37599665	0.23277465	0.04845807	0.388684458	0.51746853	0.47605935	0.4137702	0.23725225	0.385883154	0.39530379	0.53625647	0.37437458	0.41455912	0.38369278	0.36632011	0.34257523	1	0.5211475	0.38618567	0.20228122	0.3562233	0.29298621	0.58705991	0.53604946	0.38237881	0.31510313	0.45077393	0.205515
D05.SI	-0.023954405	0.40098622	0.41057872	0.2596309	0.07637871	0.457223839	0.73183903	0.59590919	0.50311068	0.29787573	0.438439589	0.47972136	0.73723373	0.48016237	0.5477585	0.42954732	0.36066106	0.378241423	0.5211475	1	0.39739437	0.23786912	0.40933593	0.39673301	0.61388816	0.62066199	0.44063076	0.34557316	0.50967527	0.2156383
F34.SI	0.028706065	0.25809527	0.42843307	0.25771458	0.04328505	0.347825682	0.4143472	0.49064251	0.35646697	0.27328861	0.320286583	0.37509965	0.43271316	0.33900772	0.44026466	0.32730431	0.3066901	0.293783001	0.38618567	0.39739437	1	0.20946408	0.30525402	0.27424416	0.41861795	0.41510952	0.29219089	0.24830059	0.40149553	0.21781919
J36.SI	0.033538775	0.1834527	0.21457463	0.14403482	0.05911279	0.19111988	0.21664361	0.28494713	0.18469909	0.6036294	0.18559713	0.31211286	0.23977154	0.13996073	0.26556809	0.21065654	0.23771698	0.153685486	0.20228122	0.23786912	0.20946408	1	0.14050191	0.1564825	0.25667792	0.25740401	0.26353463	0.14262071	0.24719128	0.14193981
G13.SI	-0.019902636	0.31098893	0.36905984	0.20691614	0.06702228	0.398148306	0.39372695	0.42091553	0.33062188	0.18902306	0.295543084	0.31073963	0.41478762	0.31663354	0.37526593	0.26825193	0.27643366	0.302259003	0.3562233	0.40933593	0.30525402	0.14050191	1	0.2698137	0.35274883	0.3921239	0.27543379	0.23583255	0.33450401	0.18092212
T39.SI	-0.021031479	0.25905568	0.26144486	0.20811046	0.07761691	0.2886306	0.38985211	0.36954329	0.35288161	0.15386893	0.344052167	0.34256087	0.41837654	0.36407193	0.32937134	0.30966994	0.24498102	0.361281281	0.29298621	0.39673301	0.27424416	0.1564825	0.2698137	1	0.33715091	0.34661404	0.27482035	0.23712395	0.36217452	0.21899931
C09.SI	-0.039808854	0.33281166	0.37502287	0.23903856	0.05065339	0.426228744	0.58237862	0.54365724	0.48955387	0.28765028	0.419805595	0.46003652	0.61554086	0.40480005	0.50096289	0.43852925	0.36459753	0.342656602	0.58705991	0.61388816	0.41861795	0.25667792	0.35274883	0.33715091	1	0.58960239	0.40318022	0.33824096	0.51984723	0.2333011
S68.SI	-0.025850939	0.36322738	0.43454705	0.24499734	0.07917218	0.485289194	0.5953527	0.56360353	0.47666378	0.28343214	0.411909987	0.47045363	0.60466304	0.40296097	0.52259791	0.48568479	0.35997059	0.39085182	0.53604946	0.62066199	0.41510952	0.25740401	0.3921239	0.34661404	0.58960239	1	0.44429777	0.33146185	0.51669295	0.23237872
C38U.SI	-0.068849041	0.27043672	0.29909444	0.17341389	0.06962906	0.261834879	0.43132723	0.40683629	0.3618681	0.28447106	0.302149221	0.39784013	0.4509628	0.34203842	0.37455918	0.53005783	0.32469149	0.337032542	0.38237881	0.44063076	0.29219089	0.26353463	0.27543379	0.27482035	0.40318022	0.44429777	1	0.41171456	0.4141073	0.21930346
C61U.SI	-0.034574915	0.30283243	0.24476941	0.17039896	0.07473872	0.230730459	0.32108939	0.31047079	0.26820728	0.19547268	0.284159104	0.31895202	0.34313138	0.29364786	0.31229395	0.39737168	0.27110137	0.254338434	0.31510313	0.34557316	0.24830059	0.14262071	0.23583255	0.23712395	0.33824096	0.33146185	0.41171456	1	0.32118146	0.17985194
U14.SI	-0.01365725	0.3261733	0.35168081	0.25570784	0.10227041	0.408958752	0.52803248	0.47130496	0.42431783	0.27837935	0.411469833	0.43042803	0.54606635	0.39279021	0.4605627	0.40434492	0.36181661	0.403505616	0.45077393	0.50967527	0.40149553	0.24719128	0.33450401	0.36217452	0.51984723	0.51669295	0.4141073	0.32118146	1	0.22371018
CC3.SI	0.03049565	0.15030383	0.15619732	0.07791024	0.02102888	0.198558174	0.22556079	0.26631419	0.20026937	0.18909735	0.23840084	0.24528196	0.25593512	0.27307262	0.23054038	0.20958946	0.18855114	0.225824962	0.205515	0.2156383	0.21781919	0.14193981	0.18092212	0.21899931	0.2333011	0.23237872	0.21930346	0.17985194	0.22371018	1