Below is my attempt to describe the method I use to get at HMIS data. In short, I’m mixing two powerful data languages to get answers from a data set quickly.
What is SQL?
SQL stands for Structured Query Language. Which can be translated as, “Asking a question a computer could understand.” This computer language was designed to get data off a remote relational database.
A sample of what SQL looks like:
SELECTDataFROMDataSetWHEREData='BlueEyes'
SQL Pros:
Easy to understand
Can be learned quickly
Powerful in merging data sets
SQL Cons:
No advanced functions
Does not allow for additional features to be added by users
R is a programming language for super nerds. Often, it is used by those working in:
Bio-statistic (genetic sequencing)
High-level economics
Big-data (think all data collected by Facebook)
Machine learning
It is extremely fast computationally. The people who designed it wanted something which could crunch big data sets, on their computer, in a matter of minutes.
One of the neatest parts of R is it allows for user written plugins. Want to translate a data set of ZIP codes into their corresponding GPS coordinateds. Their’s a plugin for that! And did I mention it’s all free!
Think of our approach to data as Spanglish. We are going to take two major programming languages to get the best of both. When we don’t have the exact word to describe something, we will switch to a different language and use its vocabulary.
R is a powerful tool, but often, the syntax is boggish. It is hard to read and figure out what’s going on. SQL on the other hand, it’s pretty simple. And humans have an easier time reading it.
Personally, I’m looking to solve problems as quickly as possible and I’ve found by mixing the two I get to solutions quick. If I’m having a hard time getting Google to tell me how to do something in SQL, I’ll switch and see if Google know’s how to do it in R. Thus, getting to what I need do much quicker. And let’s not fool ourselves–learning a programming language is learning how to Google well.
A second reason to mix SQL is about respect and marketability. R seems to be gaining ground in a lot of data sciences, and seems to be the tool when it comes to economics and statistics, however, most data exchanges have SQL at their heart. Therefore, when I can use my work as an excuse to develop a marketable skill, I’m going to do it.
A third reason to mix the two, most often the servers where our data calculations are being done are extremely limited in their hardware. This results in calculations taking much longer on a server than they would on or personal PC. For example, a query which would take 45 minutes on our HMIS software vendor’s server takes around 30 seconds on my personal PC.
A fourth reason, by completing our calculations on our personal computers it reduces the number of times client level information would need to be transmitted back-and-forth from the server. Of course, this doesn’t mean the data is more secure, it simply means the opportunities for more secure data are in our hands. This method makes follow proper PC security practices much more important (encrypting hard-drives, not sharing passwords, etc)
Localization of data.
For a long time now I’ve put off learning JavaScript. It really never interested me. I’d like to say it was the thought, “Oh, JavaScript is for web developers and I’m embedded all the way!” But that wasn’t really it. I think it hasn’t appealed to me because I couldn’t connect it to hardware. Well, at least, that was my assumption.
However, I’ve recently discovered Google’s Web APIs. Specifically, their Bluetooth Low Energy API.
It’s pretty amazing. It allows a developer to write asynchronous JavaScript using Promises to get into the hardware of the client’s PC, all from the browser!
Now, this might sound like it open for security issues–and perhaps it will be. But there are two requirements Google has put in place which hopefully gets around any issues. First, the API can only be called by action. Secondly, the API can only be called from a secured connection (HTTP over SSL).
Ok, there are few other downers to talk about. First this only works in Chrome–but given this is a Google API, well, duh. The other is not all OSes are currently supported. The following I’ve tested and work right out of the box:
Mac OS
Android
The others which are supposed to be supported but I’ve not tested:
Linux
Windows (with some work)
Chromium
Having worked with Bluetooth LE on all of these OSes I can say there is hope for Windows. In fact, I think with the Creator’s Update the Microsoft folk opened up the last needed ingredient. The real hold out will be iOS. Apple is not a fan of browser apps. They would much rather browsing be done inside their native apps. If I’m being positive, I’d say this is so Apple can make sure the mobile UX is excellent, and by forcing native apps, they have control by app approval. If I’m being negative, well, Apple takes 30% on app purchases and web apps land them nada. Just my opinion.
If you’d like to stay up to date on compatibility of BLE in the browser there is a an implementation status page on the Web Bluetooth Community Group:
The documentation is actually pretty robust–and with this guide, the process of subscribing to a device characteristic should be pretty straight forward.
The first piece we need are service IDs to search for.
This takes the text element of the DOM element ‘optionalServices’, which should be in the in 16 bit hex format, 0x0000. This becomes one of the service IDs searched in the Bluetooth LE search cycle. For the Bluetooth module HM-10, HM-11, HM-16, HM-17 the service ID is 0xFFE0.
Moving on to the search, when the code below is executed the Chrome browser should show a search and pair menu (see image) for pairing a device. When a device has been paired the promise will resolve returning the device which has been paired.
It is important to note this block must be called by a user action. For example, if set to execute on page load it will refuse to fire. But if called onClick then it will show. This is meant to provide more security to the API.
As stated, the requestDevice will return a device. Using the promise .then we can begin working with the BluetoothDevice
Which is returned after it has been paired by the user. The BluetoothDevice object has three items of interest.
name – which provides the string name of the device
id – the ID string
gatt – a gatt which contains a reference to the BluetoothRemoteGATTServer object
The BluetoothRemoteGATTServer interface contains many of the methods needed to interact with the Bluetooth device. For example,
device.gatt.connect()
Attempts to asynchronously create a connection with the device through a Promise. If .then is attached then the method will return a service object if succesful. If you are just looking to get something done with Bluetooth, feel free to keep hacking through this article (that’s what I’d do–TL;DR). However, if you want to know more about Bluetooth 4 protocol here a few useful links:
Once the connection attempt has been made and returned succesful, the BluetoothRemoteGATTServer object returned can be petitioned for a list of services.
This will fire asynchronously using promises, and if succesful, return a BluetoothRemoteGATTService object. This represents all the services the device has public. Then, the returned service object may be iterated over to identify get characteristics of the device. (Almost to the data, I swear).
Essentially, the BluetoothRemoteGATTService object is merely an array containing on the services. Using a services.forEach we get each individual service to explore its characteristics.
Now, I’m going to add the whole block which is responsible for iterating over each service and its characteristics, essentially turning on notifications for each device. This will ultimately allow a callback to be fired every every time the device sends data and a reference to a method by which data can be written to the device.
....letqueue=Promise.resolve();queue=queue.then(_=>service.getCharacteristics().then(characteristics=>{characteristics.forEach(characteristic=>{writeCharacteristic=characteristic;writeCharacteristic.startNotifications();resolve();});// End enumerating characteristics}));// End queue})// End enumerating services})// End Service exploration
The queue is a promise which allows us to loop through services and characteristics without breaking asynchronousity. Since this is my first JavaScript program, I will not try to explain it, but here’s another guy’s article which attempts to explain it:
Essentially, each service and characteristic contained in the service enumerated. At each characteristic there are two calls. One is to get a reference to the characteristic for writing. This is the global variable writeCharacteristic. Then, notifications for the writeCharacteristic are started. This will assure any time data is made available on the remote device our program is notified.
Now, it should be noted, this above code is hackish. For example, what if there are multiple characteristics and the last one isn’t the one we want to know about. Well, we’d have a write reference to the wrong characteristic. So, filtering to the desired characteritic is on my TODO list.
But, let’s finish before refactoring.
Let’s take a look at how to write data to the device after getting a reference to the desired characteristic.
varwrite=function(data,string=true){p=newPromise(function(resolve,reject){// See if the device is paired.if(pairedDevices){// Has a write reference been discovered.if(writeCharacteristic!=null){// Don't double encode.if(string){letencoder=newTextEncoder('utf-8');writeCharacteristic.writeValue(encoder.encode(data));}else{dataInUint8=Uint8Array.from(data);writeCharacteristic.writeValue(dataInUint8);}resolve();}else{reject("No write characteristic")}}else{reject("No devices paired.")}}).catch(error=>{});returnp;}
The above method creates a promise and writes to the device asynchoronously. On the way, it checks to make sure the device is paired (not connected, that’s on the TODO list). Also, it makes sure we still have a reference to the writeCharacteristic. Then, it will either encode it in utf-8 and write the data, or if the string argument is false it’ll just write the data. After it has written the data, the resolve is executed. This would allow the writeMethod to be called like so:
write("Buggers",true).then(_=>{// Do something after write has completed.})
Ok, last bit. Let’s setup capturing incoming data. To begin, I created a method which holds a list of all the callback methods to call when data has been received.
varonReceivedDataCallbacks=[];...// Adds a function called when a BLE characteristic changes value.// Mutiple callbacks may be added.this.addReceivedDataCallback=function(callback){if(writeCharacteristic){writeCharacteristic.addEventListener('characteristicvaluechanged',callback);onReceivedDataCallbacks.push({key:callback.name,value:callback})}}
This method allows a method’s name to be passed in. It then adds an event listener to this method, which will be called whenever characteristicvaluechanged. Also, it saves this method’s name in an array in case I want to stop notifications later (again, not completed, but on the TODO).
The purpose of allowing multiple callbacks is for when I’m working with many modules which all would like to know what’s going on with the Bluetooth LE device.
For example, this module is meant to be a piece of a larger project, which is an uploader app using BLE to upload HEX files to AVRs running TinySafeBoot.
Ok, one last piece. Let us see what the onRecievedData callback could looks like:
this.onReceivedData=function(event){// TODO: Handle received data better. // NOTE: the TX buffer for the HM-1X is only 20 bytes. // But other devices differ.varreceivedData=newUint8Array(event.target.value.byteLength);for(vari=0;i<event.target.value.byteLength;i++){receivedData[i]=event.target.value.getUint8(i);}}
This is how I’ve written the notification of data callback. The event.target.value contains the data, which is in an untyped array. I choice to encode it into Uint8 as I’ll be working with both ASCII and non-ASCII data.
Well, that’s it. This code will allow one to search, connect, write data to, and receive data from Bluetooth Low Energy devices from Chrome browser. Let me know if you have any recommendations.
Here is the full code referenced directly from my project:
varLumiBluetooth=(function(){// PrivatesvarpairedDevices={};varonReceivedDataCallbacks=[];varwriteCharacteristic;varwriteBuffer=[];varwriting=false;varnapsSinceWrite=0;// Adds a function called when a BLE characteristic changes value.// Mutiple callbacks may be added.this.addReceivedDataCallback=function(callback){if(writeCharacteristic){writeCharacteristic.addEventListener('characteristicvaluechanged',callback);onReceivedDataCallbacks.push({key:callback.name,value:callback})}}// Clears the RecievedDataCallback dictionary.this.removeAllReceivedDataCallbacks=function(){onReceivedDataCallbacks=[];}// Searches for Devices based upon Service IDs. Then prompts// a user to select a target device. Lastly, it conencts to// target d evice.this.searchAndConnect=function(primaryServicesUUID,addSystemText=""){returnnewPromise(function(resolve,reject){letoptionalServices=document.getElementById('optionalServices').value.split(/, ?/).map(s=>s.startsWith('0x')?parseInt(s):s).filter(s=>s&&BluetoothUUID.getService);if(addSystemText){addSystemText('Requesting any Bluetooth Device...');}navigator.bluetooth.requestDevice({acceptAllDevices:true,optionalServices:optionalServices})// After getting a device.then(device=>{pairedDevices[device.name]=device;if(addSystemText){addSystemText('Connecting to GATT Server...');}returndevice.gatt.connect();})// After connecting.then(server=>{if(addSystemText){addSystemText('Getting Services...');}returnserver.getPrimaryServices();})// After getting services.then(services=>{if(addSystemText){addSystemText("Found services: ");}services.forEach(service=>{letqueue=Promise.resolve();queue=queue.then(_=>service.getCharacteristics().then(characteristics=>{if(addSystemText){addSystemText('Service: '+service.uuid);}characteristics.forEach(characteristic=>{if(addSystemText){addSystemText('>> Characteristic: '+characteristic.uuid+''+getSupportedProperties(characteristic));}writeCharacteristic=characteristic;if(addSystemText){addSystemText("Write characteristic set");}writeCharacteristic.startNotifications();resolve();});// End enumerating characteristics}));// End queue})// End enumerating services}).// End Service exploration catch(error=>{if(addSystemText){addSystemText(error);}})});// End Search and Connect Promise}// End Search and Connect Functionthis.writeString=asyncfunction(data,addSystemText=null){write(data,true,addSystemText);}this.writeData=asyncfunction(data,addSystemText=null){write(data,false,addSystemText);}varwrite=function(data,string=true,addSystemText=null){p=newPromise(function(resolve,reject){if(pairedDevices){if(writeCharacteristic!=null){// Don't double encode.if(string){letencoder=newTextEncoder('utf-8');varwriteData=encoder.encode(data);writeBuffer.push.apply(writeBuffer,writeData);// test// writeBuffer = appendUint8Buffer(writeBuffer, writeData);writeLoop(writeData);}elseif(data!=null){writeData=Array.from(data);// dataInUint8 = Uint8Array.from(data);writeBuffer.push.apply(writeBuffer,writeData);// writeBuffer = appendUint8Buffer(writeBuffer, dataInUint8);writeLoop(writeData);}else{resolve();}resolve();}else{reject("No write characteristic")}}else{reject("No devices paired.")}}).catch(error=>{if(addSystemText){addSystemText("No device paired");}});returnp;}this.disconnectDevice=function(){}// Important information on write queue// https://github.com/WebBluetoothCG/web-bluetooth/issues/188// varwriteLoop=asyncfunction(data){// writing = true;// for(var i = 0; i < writeBuffer.length; i){// var length = 0;// // if(writeBuffer.length < (i + 20)){ length = writeBuffer.length} else { length = i + 20; }// if(writeBuffer.length < 20){ length = writeBuffer.length; } else { length = 20; }// var tmpWriteBfr = Uint8Array.from(writeBuffer.splice(0, length));// console.log(tmpWriteBfr);// writeCharacteristic.writeValue(tmp=WriteBfr);// await sleep(42);// i+=20;// }// 1. Create a rollback buffer, in case there is an error writing.// 2. Check if the buffer is empty.// 3. If writing is currently in progress, wait. But not forever.// After three naps, assume write is complete.// 4. Limit the write to the HM-10 TX buffer (20 bytes)// TODO Make the TX buffer size mutable. For example, the // HM-16 has a TX buffer of 256// 5. Cut a chunk off the writeBuffer for writing.// 6. Attempt to write the value to the device// 7. Once the write is complete, set the writing flag to false.// NOTE The write callback doesn't seem to be working.// 8. If there is an error restore the buffer.varmementoWriteBuffer=[];while(writeBuffer.length>0){if(writing===false){writing=true;varlength=0;if(writeBuffer.length<20){length=writeBuffer.length;}else{length=20;}mementoWriteBuffer=writeBuffer;vartmpWriteBfr=Uint8Array.from(writeBuffer.splice(0,length));console.log("TX: "+tmpWriteBfr);writeCharacteristic.writeValue(tmpWriteBfr).then(blah=>{writing=false;}).catch(error=>{writeBuffer=mementoWriteBuffer;writing=false;console.log("BLE Write Error: ");console.log(error);delayAndWriteAgain();});}else{awaitbleWriteThrottling(42);}}writeBuffer=[];}vardelayAndWriteAgain=function(){setTimeout(writeLoop(),100);}varbleWriteThrottling=asyncfunction(ms){// 1. Sleep a bit// 2. Count naps// 3. Too many naps, then assume BLE writing is done.awaitsleep(ms);napsSinceWrite++;if(napsSinceWrite>2){writing=false;napsSinceWrite=0;}}/* Utils */functiongetSupportedProperties(characteristic){letsupportedProperties=[];for(constpincharacteristic.properties){if(characteristic.properties[p]===true){supportedProperties.push(p.toUpperCase());}}return'['+supportedProperties.join(', ')+']';}varappendUint8Buffer=function(bufferOne,bufferTwo){if(!bufferOne){returnbufferTwo;}vartmp=newUint8Array(bufferOne.byteLength+bufferTwo.byteLength);tmp.set(newUint8Array(bufferOne),0);tmp.set(newUint8Array(bufferTwo),bufferOne.byteLength)returntmp.buffer;}return{addReceivedDataCallback:addReceivedDataCallback,searchAndConnect:searchAndConnect,writeString:writeString,writeData:writeData,disconnectDevice:disconnectDevice}})();// End Proto
This is an R script which will take two sets of HMIS 5.1 CSVs and produce a combined set.
A few notes:
A new ExportID will need to be provided.
Each files are deduplicated based upon the Primary Key (ProjectEntryID, PersonalID, etc.)
The Project.csv contains a PITCount which is different based upon the date ranges the two data sets were pulled. However, the script takes the maximum of the two PITCounts.
It requires HMIS_Functions and dplyr.
library(dplyr)# hmisFunctions <- "/Users/user//Dropbox/HMIS/HMIS_R_Functions/HMIS_R_Functions.R"# dataPathOne <- "/Users/user//Dropbox/HMIS/R HMIS CSV Set Merger/HMIS Data 10-01-2016 to 10-31-2016"# dataPathTwo <- "/Users/user//Dropbox/HMIS/R HMIS CSV Set Merger/HMIS Data 11-01-2016 to 11-30-2016"# pathForCombinedData <- "/Users/user//Dropbox/HMIS/R HMIS CSV Set Merger/"# nameOfMergedDirectory <- "Merged"hmisFunctions<-"/Users/user/Dropbox/HMIS/HMIS_R_Functions/HMIS_R_Functions.R"dataPathOne<-"/Users/user/Dropbox/HMIS/R HMIS CSV Set Merger/HMIS Data 10-01-2016 to 10-31-2016"dataPathTwo<-"/Users/user/Dropbox/HMIS/R HMIS CSV Set Merger/HMIS Data 11-01-2016 to 11-30-2016"pathForCombinedData<-"/Users/user/Dropbox/HMIS/R HMIS CSV Set Merger/Merged/"nameOfMergedDirectory<-"Merged"newExportID<-"12345"source(hmisFunctions)dir.create(file.path(pathForCombinedData,nameOfMergedDirectory),showWarnings=FALSE)setwd(file.path(pathForCombinedData,nameOfMergedDirectory))################################### Merge Functions ###################################mergeHmisCsvsWithUniqueIDAndExport<-function(df1,df2,uniqueIDHeader,exportID){# Merge the datamergedDf<-rbind(df1,df2)mergedDf[,uniqueIDHeader]<-as.factor(mergedDf[,uniqueIDHeader])# Drop columns which would resist removing duplicatesdrops<-c(uniqueIDHeader,"ExportID")mergedDf<-mergedDf[,!(names(mergedDf)%in%drops)]# Get only unique recordsmergedDf<-unique(mergedDf)# Creat a PrimaryKey vector.xName<-rownames(mergedDf)# Add Primary Key back.mergedDf<-cbind(xName=xName,mergedDf)# Create an ExportID vector.exportIDVector<-rep(exportID,length(mergedDf$xName))# Add ExportID vector back.mergedDf<-cbind(mergedDf,exportIDVector)# Rename newly created PrimaryKey and ExportID appropriately.colnames(mergedDf)[ncol(mergedDf)]<-"ExportID"colnames(mergedDf)[1]<-uniqueIDHeadermergedDf}mergeHmisCsvsWithExportAndPrimaryKey<-function(df1,df2,primaryKey,exportID){# Merge the datamergedDf<-rbind(df1,df2)# Drop columns which would resist removing duplicatesdrops<-c("ExportID")mergedDf<-mergedDf[,!(names(mergedDf)%in%drops)]# Get only unique records#mergedDf <- unique(mergedDf, incomparables = FALSE)mergedDf<-mergedDf[!duplicated(mergedDf[,c(primaryKey)]),]# Create an ExportID vector.exportIDVector<-rep(exportID,length(mergedDf[1]))# Add ExportID vector back.mergedDf<-cbind(mergedDf,exportIDVector)# Rename newly created PrimaryKey and ExportID appropriately.colnames(mergedDf)[ncol(mergedDf)]<-"ExportID"mergedDf}mergeHmisCsvsWithExportId<-function(df1,df2,exportID){# Merge the datamergedDf<-rbind(df1,df2)# Drop columns which would resist removing duplicatesdrops<-c("ExportID")mergedDf<-unique(mergedDf[,!(names(mergedDf)%in%drops)])# Create an ExportID vector.exportIDVector<-rep(exportID,length(mergedDf[1]))# Add ExportID vector back.mergedDf<-cbind(mergedDf,exportIDVector)# Rename newly created PrimaryKey and ExportID appropriately.colnames(mergedDf)[ncol(mergedDf)]<-"ExportID"mergedDf}###################### Merge Affiliation ######################affiliationOne<-loadAffiliation(dataPathOne)affiliationTwo<-loadAffiliation(dataPathTwo)affiliationCombined<-rbind(affiliationOne,affiliationTwo)affiliationCombined<-unique(affiliationCombined)remove(list=c("affiliationOne","affiliationTwo"))###################### Merge Client ######################clientOne<-loadClient(dataPathOne)clientTwo<-loadClient(dataPathTwo)clientCombined<-rbind(clientOne,clientTwo)clientCombined<-unique(clientCombined)remove(list=c("clientOne","clientTwo"))####################### Merge Disabilities #######################disabilitiesOne<-loadDisabilities(dataPathOne)disabilitiesTwo<-loadDisabilities(dataPathTwo)disabilitiesCombined<-mergeHmisCsvsWithUniqueIDAndExport(disabilitiesOne,disabilitiesTwo,"DisabilitiesID",newExportID)remove(list=c("disabilitiesOne","disabilitiesTwo"))################################### Merge Employment and Education ###################################employmentEducationOne<-loadEmployementEducation(dataPathOne)employmentEducationTwo<-loadEmployementEducation(dataPathTwo)employmentEducationCombined<-mergeHmisCsvsWithUniqueIDAndExport(employmentEducationOne,employmentEducationTwo,"EmploymentEducationID",newExportID)remove(list=c("employmentEducationOne","employmentEducationTwo"))###################### Merge Enrollment ######################enrollmentOne<-loadEnrollment(dataPathOne)enrollmentTwo<-loadEnrollment(dataPathTwo)enrollmentCombined<-mergeHmisCsvsWithExportAndPrimaryKey(enrollmentOne,enrollmentTwo,"ProjectEntryID",newExportID)remove(list=c("enrollmentOne","enrollmentTwo"))######################## Merge EnrollmentCoC ########################enrollmentCocOne<-loadEnrollmentCoc(dataPathOne)enrollmentCocTwo<-loadEnrollmentCoc(dataPathTwo)enrollmentCocCombined<-mergeHmisCsvsWithUniqueIDAndExport(enrollmentCocOne,enrollmentCocTwo,"EnrollmentCoCID",newExportID)remove(list=c("enrollmentCocOne","enrollmentCocTwo"))###################### Merge Exit ######################exitOne<-loadExit(dataPathOne)exitTwo<-loadExit(dataPathTwo)exitCombined<-mergeHmisCsvsWithExportAndPrimaryKey(exitOne,exitTwo,"ExitID",newExportID)remove(list=c("exitOne","exitTwo"))###################### Merge Expot ######################exportOne<-loadExport(dataPathOne)exportTwo<-loadExport(dataPathTwo)exportCombined<-exportTworemove(list=c("exportOne","exportTwo"))###################### Merge Funder ######################funderOne<-loadFunder(dataPathOne)funderTwo<-loadFunder(dataPathTwo)funderCombined<-mergeHmisCsvsWithExportId(funderOne,funderTwo,newExportID)remove(list=c("funderOne","funderTwo"))###################### Merge Health & DV ######################healthAndDVOne<-loadHealthAndDv(dataPathOne)healthAndDVTwo<-loadHealthAndDv(dataPathTwo)healthAndDVCombined<-mergeHmisCsvsWithUniqueIDAndExport(healthAndDVOne,healthAndDVTwo,"HealthAndDVID",newExportID)remove(list=c("healthAndDVOne","healthAndDVTwo"))############################## Merge Income and Benefits ##############################incomeBenefitsOne<-loadIncomeBenefits(dataPathOne)incomeBenefitsTwo<-loadIncomeBenefits(dataPathTwo)incomeBenefitsCombined<-mergeHmisCsvsWithUniqueIDAndExport(incomeBenefitsOne,incomeBenefitsTwo,"IncomeBenefitsID",newExportID)remove(list=c("incomeBenefitsOne","incomeBenefitsTwo"))###################### Merge Inventory ######################inventoryOne<-loadInventory(dataPathOne)inventoryTwo<-loadInventory(dataPathTwo)inventoryCombined<-mergeHmisCsvsWithUniqueIDAndExport(inventoryOne,inventoryTwo,"InventoryID",newExportID)remove(list=c("inventoryOne","inventoryTwo"))####################### Merge Organization #######################organizationOne<-loadOrganization(dataPathOne)organizationTwo<-loadOrganization(dataPathTwo)organizationCombined<-mergeHmisCsvsWithExportId(organizationOne,organizationTwo,newExportID)remove(list=c("organizationOne","organizationTwo"))###################### Merge Project ######################projectOne<-loadProject(dataPathOne)projectTwo<-loadProject(dataPathTwo)projectsCombined<-rbind(projectOne,projectTwo)# Get only the highest PIT CountprojectsCombined<-projectsCombined%>%group_by(ProjectID)%>%filter(PITCount==max(PITCount))# Remove ExportID column for flatteningdrops<-c("ExportID")projectsCombined<-projectsCombined[,!(names(projectsCombined)%in%drops)]projectsCombined<-unique(projectsCombined)# Create an ExportID vector.exportIDVector<-rep(newExportID,length(projectsCombined$ProjectID))# Add ExportID vector back.projectsCombined<-as.data.frame(projectsCombined)projectsCombined<-cbind(projectsCombined,exportIDVector)colnames(projectsCombined)[ncol(projectsCombined)]<-"ExportID"remove(list=c("projectOne","projectTwo"))###################### Merge Project CoC ######################projectsCoCOne<-loadProjectCoc(dataPathOne)projectsCoCTwo<-loadProjectCoc(dataPathTwo)projectCoCCombined<-rbind(projectsCoCOne,projectsCoCTwo)# Get only the highest PIT CountprojectCoCCombined<-projectCoCCombined%>%group_by(ProjectID)%>%filter(PITCount==max(PITCount))# Remove ExportID column for flatteningdrops<-c("ExportID")projectCoCCombined<-projectCoCCombined[,!(names(projectCoCCombined)%in%drops)]projectCoCCombined<-unique(projectCoCCombined)# Create an ExportID vector.exportIDVector<-rep(newExportID,length(projectCoCCombined$ProjectID))# Add ExportID vector back.projectCoCCombined<-as.data.frame(projectCoCCombined)projectCoCCombined<-cbind(projectCoCCombined,exportIDVector)colnames(projectCoCCombined)[ncol(projectCoCCombined)]<-"ExportID"remove(list=c("projectsCoCOne","projectsCoCTwo"))###################### Merge Services ######################servicesOne<-loadServices(dataPathOne)servicesTwo<-loadServices(dataPathTwo)servicesCombined<-mergeHmisCsvsWithUniqueIDAndExport(servicesOne,servicesTwo,"ServicesID",newExportID)remove(list=c("servicesOne","servicesTwo"))###################### Merge Site ######################siteOne<-loadSite(dataPathOne)siteTwo<-loadSite(dataPathTwo)siteCombined<-mergeHmisCsvsWithExportId(siteOne,siteTwo,newExportID)remove(list=c("siteOne","siteTwo"))############################# Write combined HMIS CSVs #############################write.csv(affiliationCombined,file=paste(pathForCombinedData,"Affiliation.csv",sep=""),na="",row.names=FALSE)write.csv(clientCombined,file=paste(pathForCombinedData,"Client.csv",sep=""),na="",row.names=FALSE)write.csv(disabilitiesCombined,file=paste(pathForCombinedData,"Disabilities.csv",sep=""),na="",row.names=FALSE)write.csv(employmentEducationCombined,file=paste(pathForCombinedData,"EmploymentEducation.csv",sep=""),na="",row.names=FALSE)write.csv(enrollmentCombined,file=paste(pathForCombinedData,"Enrollment.csv",sep=""),na="",row.names=FALSE)write.csv(enrollmentCocCombined,file=paste(pathForCombinedData,"EnrollmentCoC.csv",sep=""),na="",row.names=FALSE)write.csv(exitCombined,file=paste(pathForCombinedData,"Exit.csv",sep=""),na="",row.names=FALSE)write.csv(exportCombined,file=paste(pathForCombinedData,"Export.csv",sep=""),na="",row.names=FALSE)write.csv(funderCombined,file=paste(pathForCombinedData,"Funder.csv",sep=""),na="",row.names=FALSE)write.csv(healthAndDVCombined,file=paste(pathForCombinedData,"HealthAndDV.csv",sep=""),na="",row.names=FALSE)write.csv(incomeBenefitsCombined,file=paste(pathForCombinedData,"IncomeBenefits.csv",sep=""),na="",row.names=FALSE)write.csv(inventoryCombined,file=paste(pathForCombinedData,"Inventory.csv",sep=""),na="",row.names=FALSE)write.csv(organizationCombined,file=paste(pathForCombinedData,"Organization.csv",sep=""),na="",row.names=FALSE)write.csv(projectsCombined,file=paste(pathForCombinedData,"Project.csv",sep=""),na="",row.names=FALSE)write.csv(projectCoCCombined,file=paste(pathForCombinedData,"ProjectCoC.csv",sep=""),na="",row.names=FALSE)write.csv(servicesCombined,file=paste(pathForCombinedData,"Services.csv",sep=""),na="",row.names=FALSE)write.csv(siteCombined,file=paste(pathForCombinedData,"Site.csv",sep=""),na="",row.names=FALSE)
One of the issues I’ve had in the past with the Lumi projects is manageable UI. The project will start out pretty straight foward, but soon, I’m switching between device types, checking if hardware is ready, and routing callbacks based upon the device selected. It becomes spaghetti code quick. On Lumi4, I’ve decided to bite the bullet and implement MVVM.
After about 20 hours struggling with setting up Lumi4 as an MVVM project I’ve dervied two conclusions:
It’s possible
Apple spoils developers with MVC baked into Xcode
MVVM in C# and UWP isn’t simple. It seems like there is a lot more boiler-plate coding necessary for MVVM than MVC in Xcode. Eventually, I broke down and downloaded the NuGet package Prism, as a MVVM helper. This helped alleviate some of the code necesssary for Commanding (which I still don’t understand well enough to implement without Prism).
Below I’m going to take a look at a couple of controls I’ve written MVVM on. Finding documentation for MVVM in Universal Windows Plateform (UWP) is tricky. It is similar to XamarinForms and WPF, but overall, there are syntax differences which make generalizing the documentation difficult.
This sets up the binding to the variables HostIDOne, HostIDTwo, NetworkIDOne, NetworkIDTwo. However, there’s plenty more boilerplate code before things start working.
To setup a ViewModel it’s best to setup an abstract class which can be inherited. This saves on creating boilerplate. Below is the abstract class the internet told me to make:
This code handles the property changing notification for all properties declared in classes which inherit from the MainViewModelBase. Such as the MainViewViewModel which is used in Lumi4
Moving on to the actual implementation of the bound text boxes. Each text box will have a property associated with the string value in the Text attribute. However, before this will work, the DataContext must be set for the MainPage.xaml. This is done in the MainPage.xaml.cs file.
The if statement under the setter checks if the value about to be set is the same as the value is currently. Note, there is a helper in Prism which will prevent one from having to rewrite this for every attribute. However, I implemented the above code before I downloaded Prism and decided to write this article while it was still fresh in my mind. I’ll probably correct these to use the Prism helper before moving.
The code so far should allow for the properties setter to be called whenever the user types something in one of the four textboxes. On caveat, it seems the setter is not called until the user removes focus from the textbox. I’ll probably need to change the binding of the properties, but for now, it works well enough.
Well, this is all fine and gone, but what about Commands such as Button Click Events? Pretty simple, but with more boilerplate code.
Next, and this is the part I don’t understand without Prism helping, there needs to be a delegate which will fire an event whenever the commad is called. In Prism there is the DelegateCommand type which takes care of a lot of the work. The DelegateCommand has to be initialized with two event handlers CanExecute and Execute. These methods will be called in that order every time the DelegateCommand property is accessed.
And that’s it for now. Had to make sure I jotted down my notes while the challenge was still fresh in my head.
Update: 4-16-2017
Apparently, a Prism ViewModel should inherit from BindableBase instead of the custom ViewModelBase the internet told me to write. Also, to get the prismProp code snippet it looks like you have to download the Prism template packet:
