Friday, September 23, 2016

ProsperWarehouse - Building less-bad code

EVE Prosper is first and foremost a data science project.  And though hack-and-slash has got us this far, we need to consider a proper design/environment if we want to actually expand coverage rather than just chase R/CREST/SQL bugs.


There has been some work moving Prosper to a v2 codebase (follow the new github projects here) but ProsperWarehouse is a big step toward that design.  This interface should allow us to open up a whole new field of projects, so it's critical nail this design on the first-pass before moving on.

What The Hell Is This Even
Building a Database Abstraction Layer (DAL).  

Up to now we have used ODBC, but there are some issues with cross-platform deployment, and database-specific weirdness that have caused issues: such as ARM and MacOS support being painful.  Furthermore, relying only on ODBC means we aren't able to integrate non-SQL sources like MongoDB or Influx into our stack without rewriting huge chunks of code.  Lastly, we have relied on raw-SQL and string-hacks sprinkled all over the original codebase, making updates a nightmare.

There are two goals of this project:
  1. Reduce complexity for other apps by giving standardized get/put methods.
  2. Allow easier conversion of datastore technologies.  Change connection without changing behavior
By adopting Pandas as the actual data-transporter, this means everything can talk the same talk and move data around with very little effort.  Though some complexity will come from cramming noSQL style data into traditional dataframes, that complexity can be abstracted under the hood and always yield the same structures when prompted.

How Does It Work?
The Magic of Abstract Methods

I've never been a great object-oriented developer, and I've been especially weak with parent/children relationships.  Recent projects at work have taught me some better tenants of API design and implementation, and I wanted to apply those lessons somewhere personal.  



Database Layer

Holds generic information about the connection; esentially the bulk of the API skeleton.  Whatever Database() defines will need to be filled in by its children.  This container doesn't do much work, but acts as the structure for the whole project under the hood.

Technology Layer

Right now, that's only SQLTable(), but this is designed to hold/init all the technology-specific weirdness.  Connections, query lingo, test infrastructure, configurations.  This is supposed to be interchangeable so you could pull out the SQLTable and replace it with a MongoDB- or Influx-specific structure.  This isn't 100% foolproof with some of the test hooks the way they are built in right now, but by standardizing input/output, conversion shouldn't be a catastrophe.

Datasource Layer

A connection-per-resource is the goal going forward.  This means we give up JOIN functionality inside SQL, but gain an easier to manage resource that can be abstracted.  All of the validation, connection setup/testing, and any special-snowflake modifications go to this layer.  Also, because these have been broken out into their own py files, debug tests can be built into __main__ as a way for humans to actually fix problems without having to rely on shoddy debug/logging.

This adds a lot of overhead for initializing a new datasource.  In return for that effort we get the ability to test/use/change those connections as needed rather than going up a layer and fixing everything that connected to that source.  It's not free, but should be a cost-benefit for faster development down the line.

Importlib Magic

The real heavy lifter for the project isn't just the API object design, but a helper that turns an ugly set of imports/inits into a far simpler fetch_data_source() call.  I would really like to dedicate a blog to this, but TL;DR: importlib lets us interact with structures more like function-pointers.  This was useful for a work project because we could execute modules by string rather than using a "main.py" structure that would need to import/execute every module in-sequence.  This should make it so you just have to import one module and get all the dependent structure automagically.

Without importlib, every datasource would have to be imported like:



Instead now it can look like


A small change, but it should clean up overhead and allow for more sources to be loaded more easily.  Also, this does mean you could fork the repo and build your own table_config path without going crazy trying to path everything.

A Lot Of Work For What Exactly?

The point is to simplify access into the databases.  With a unified design there, we can very easily lay the groundwork for a Quandl-like REST API.  Also, with the query logic simplified/unified, writing direct apps to both fetch/process the data go from 100+ lines of SQL to 2-3 lines of connection.  

By abstracting a painful piece of the puzzle, this should make collaboration easier.  This also buys us the ability to use a local-only dummy sources for testing without production data, so collaborators can run in a "headless mode".  Though I doubt I will get much assistance on updating the Warehouse code, it's a price worth paying to solve some of the more tedious issues like new cron scripts or REST API design with less arduous SQL-injection risk/test.

Monday, September 12, 2016

Fantasy Reflects Reality - August Economic Report

Thanks CCP Quant for releasing the stats AFTER the latest Prosper episode.  Monthly Economic Report - August 2016

Though there isn't anything particularly bad in this month's numbers, it didn't really live up to my expectations outlined in last month's summary.

Tell Me What You See

It's very "second verse, same as the first" looking from July into August.  A lot of the bulk stats look very good.  Net trade and PVP rates are staying steady and moving as-expected ahead of the YC118.8 release with its rebalance to mining.

Looking back at our favorite pairing, PLEX and ISK velocity, things looks good.  We're finally back over the 1B mark, and trending positive. 


PLEX is going to be extremely interesting in the coming weeks thanks to Alpha Clones.  Though I believe most of the positive and negative trends wash eachother out, I think there are some big changes in store for other trends outside of PLEX.  Specifically if more people are logged in and interacting with the game, even with the handicapped-clones, those could put some measurable pressure on mineral consumption and ISK velocities.

Also, most of the discussions I've seen around Alpha Clones center around "free ISK" and though the effects won't be zero (esp for skill trading... free baseclones), I think the pain of exploiting Alphas will tamp down most of the worst worries.  Lastly, we wouldn't be getting the F2P program if Team Security weren't up to sniffing out illicit activity, so I am not overly worried about massive botting rings.

Troubling Headwinds

What has me worried in the short term isn't so much the levels.  Absolute readings from all indicators are looking good.  But looking at the month-to-month rates have me worried.  Looking at the wallet statistics, I'm happy with the total ISK, I'm happy with the sinks/faucets, but I'm not happy about the rate of wallet growth or ISK velocity.  


I expected after WWB+Citadel that players would be able to recoup their lost wealth and start making roads toward regaining normal.  And though the sink/faucet chart says activity is returning to normal, the wallet status graph says total ISK out there to work with is diminished.  The trend isn't reason to raise alarm, but I will be watching these stats going into the winter.

Drawing Conclusions

I'm a big fan of Marketplace (and if I didn't have to work for a living, I'd produce those kinds of stories for EVE).  And if you follow their Friday Roundup, a recent theme with the IRL economic numbers are "it's good... but...".  I have a similar feeling about this month's report; I expected trends to start pointing upward into the fall, but we're just seeing flatness.  We also aren't seeing any fallout +/- from No Man's Sky, which is just a little weird given the bad press post-launch, but high engagement in a scifi property tangentially linked to EVE; I was anticipating some conversion to come back.  

The devs at CCP seem to be happy with the trends, but I remain slightly worried.  Things will most likely be totally fine, but there are some troubling headwinds in the latest numbers to just be wary of.  I fully expect to have more interesting news to report in the September and October numbers.


Friday, August 19, 2016

Switching to Sublime - #DevfleetFriday

This isn't strictly EVE related, but Talk Python To Me has inspired me to post some more dev/meta related content.

To those who have worked with me on code, you'll know at least two things:

  1. I'm a total luddite when it comes to adding helpful tools to my belt (IDE's specifically)
  2. I live and work in Windows when I can
But my new job has me working in Mac and Unix far more often, so my favorite editor Notepad++ is out.  After touring editors and IDEs for a replacement, I have settled on my new home: Sublime 

Sublime New World

"Why Sublime?" was a pretty easy question for me to answer: very cross-platform, and a deep utility library.  In fact, I've pointed my configs at dropbox so I can easily work in the same Sublime across machines.  Also, the cross-language formatting profiles are pretty slick, and was able to get things working the way I wanted without too much pain.

But Sublime really wins with its Package Control.  Nearly anything you would want can be added to the editor in just a couple commands.  And by-and-large, they're easy to set up and use right out of the box.  Also, many of the packages I've dug into are mostly python, so it's been easy to fix small bugs.

Personally, SublimeLinter has been the best thing to happen to my code recently.  Though it took some effort to get working with python3 on the Mac, having the tattle-tale highlights for bad code and bad practices has me writing far more consistent code.  Writing a .pylintrc file is a little obnoxious, and it doesn't validate virtualenv packages (only ones installed in main site-packages), once it's configured it's a real godsend for keeping my code in line.

My one knock against the platform is the need to memorize a large number of shortcuts that aren't inherently obvious.  There are some really slick tricks in the arsenal, but it feels very VIMy to have them handy

Things I Miss From N++

Search Highlighting

N++ has this amazing multi-color highlighting tool that makes traversing large or complicated code blocks easy.  Word Highlight is an okay replacement, but it doesn't do multi-highlight the way I'd like.  

Project Searching

Another very slick N++ feature is its extended search.  Able to quickly see results, contexts, and previous searches in a docked pane was another useful tool I miss having.  Sublime has an extended search, but it pops open in a new document, and doesn't easily fold the way N++ does.

Multi-Pane Interface

This is a total nitpick.  Sublime absolutely does have multi-pane views, and they are actually a step up in the aggregate.  But my one complaint is duplicating a file isn't really obvious.  The "New View Into File" command isn't mapped, and I'd rather r-click a tab and send to pane.

Free or Fee?

My one complaint about Sublime is its licensing.  Though it has a winrar-type "free forever*" mode, it complains semi-randomly at save-time to purchase a license.  At $70, I think the fee is a little steep given how much competition is out there, and its key method (RSA hasn?) is not super intuitive.  

I'm not at all against paying for a thing, but recently many enterprise services have really stepped up their game and spoiled me with their set-and-forget memberships.  Services like XSplit and Adobe just work, and though I pay pretty handsomely for the memberships in the long run, I'd almost rather buy Sublime through a Patreon model.  At the absolute least, I would like an account auth primary key, even if the current key method works as an offline backup.

Monday, August 8, 2016

The Slow Summer - July Economic Report

It's been a busy summer, despite the hiatus.  The latest Economic Report numbers were particularly weak, and I was considering skipping the month.  But MarketsForISK has been publishing some serious articles and Delonewolf over at EVE Talk posted a review.



Though my fellow pundits took some serious dives into the data, I'd like to counterpoint with a more brisk review.  Be sure to check them out if you want more depth, but let's take a more general look at the stats.

Tell Me What You See

Ohboy, though last month's review was largely positive, thanks to "better than expected" activity metrics from the Serpentis event, July crashed down hard.  The month-to-month sinks and faucets chart was particularly troubling.


Though specific ISK sink/faucet numbers are in-line with the expected baseline, the Active ISK Delta (money leaving through inactive accounts) is worrying.  June's retraction was expected (WWB + Citadel) but July keeping up the trend is what concerns me.  Pair this with my favorite stat, ISK Velocity, and we're seeing a much harder retraction than I originally expected.



With the latest Blog Banter stirring the EVE Is Dying pot, these numbers could validate those looking to catastrophize.  I would not be so hasty to eulogize though.

Finding The Light

I don't think the numbers are all bad.  Looking at the net-trade and PVP stats, things are still pretty positive.  Are they breaking any records?  No.  But the activity is high enough to to be "normal" without having to panic.

NOTE: August dip due to database issue

Most market watchers have focused on the month-to-month net trade statistics (down 10-15% each month), but I've avoided them because I think they're a bit of a red herring.  For one, Feb-May numbers are much higher-than-average due to a series of effects all running together; so cooling should be expected.  Secondly, I think just talking about total-trade isn't as useful as splitting it up.  My analysis chops up the RMT markets (which make up a significant portion of the pie) and let us focus on the pieces.  This way we can see how each is moving to color the whole picture.

Material trade continues to be a hot market.  Those that are active in the game are still getting their content.  Also, as I've been saying since the last o7 show, it's an excellent time to be generating cash.  PLEX prices have only just recovered to the 1B mark (thanks to the AT auction).  The slump in the ship trade has me worried, but looking at the PVP statistics, it's hard to figure out what exactly is going on with PVP stats staying even while ships traded falls.


Lastly, we're still not quite seeing the cash-recovery I was expecting though, so there is still a lot of work left to be done to get back to truly normal levels.

Drawing Conclusions

It's easy to catastrophize in the summer.  Numbers tend to slump most at the end of July, campaigns slow, and CCP's news crawls through July/August due to vacation time.  I still believe there's enough on the plate this fall to be excited about, that as long as people don't get too bitter, there will be content to come back to once vacations end.

Looking externally, No Man's Sky is probably going to make the August numbers particularly bad.  Though, I do expect it to have a positive effect in the longer term, reigniting the hard-scifi spark that only a few games can, and perhaps bringing some contingent back for nostalgia.

If I may dip into Blog Banter territory and editorialize: it's a terrible time for picking out trends.  With the seasonal ebb and flow is at it's lowest point, drawing a line between June/July numbers would be Fox News grade cherry picking (o/ Sion).  Though I personally share a lot of Sugar Kyle's feelings of IRL vs EVE, and have been drifting more and more away from active play into a devfleet/metagame kind of role; like Jonny Pew, I just cannot drop the game entirely.  Is EVE going into a new chapter?  Absolutely.  Dying?  MMO's are dying, but I don't think EVE is doomed yet.

Wednesday, July 13, 2016

Beating The Heat - June Econ Report

CCP Quant continues to deliver the economic data.  June data released last week, and though summer is traditionally quiet, let's see if the data bears that out.  We did release our own market summary for the o7 show, but Quant's numbers released a good week after that show



ISK Velocity

As I said in last month's review, ISK velocity is the first stat I jump to.  I have this hope that ISK velocity will be useful in building better PLEX forecasts, and it's a good market activity metric to pair against PVP numbers.  Now with the benefit of hindsight, it's a little easier to draw the correlation between ISK velocity and PLEX direction.  With things cooling to a new normal level, I'm looking forward to tempering expectations a little better going forward.


When it comes to PLEX's record slide over the spring, it's useful to picture ISK velocity and PLEX prices together.  February had skill trading release, March/April had World War Bee, May had Citadels.  In this historic period of cash demand, PLEX (like gold) acts as an easy way to get at that liquidity.  With big door-busting features calmed down for the summer, I'm happy to see ISK velocity staying strong (~0.7) vs pre-skill-trading levels.

Small Note for the stats nerds: Quant's report uses a bi-directional 30d moving average to calculate ISK velocity.  I would rather use one that only looks backwards.  Will probably tweak the stat slightly going forward, which will take us out of alignment with Quant's official charts.


When it comes to activity statistics, I'm actually surprised how well June did in aggregate.  Though we're seeing a general cooling, we're actually above the points I originally spitballed.  Specifically, the slight uptick in value destroyed is interesting.  I'm not ready to pin this entirely on the new event/opportunity system, but these are positive first metrics for a contentious feature.

Citadel Math

Though the general statistics look good, Citadels are weird.  We can't yet track them in our stats (though recent updates have cleared up the API blindspot) and a wild west of bug-or-feature has made it hard to get a solid hand around Citadels in the long term.

Rhivre has been the authority when it comes to citadels.  Specifically, she brought up that the June Faucet/Sink graph looked particularly light.  Quant has responded that Citadel data should be in the report.



The sinks retracted a lot more than many originally expected.  Also, the big dip in Active ISK Delta (money leaving the system because of account lapse) is mildly troubling.  Though April/May proved to be record setting in terms of ISK sinks, June returns us back to the ~45T level.  Half because blueprint outlays (citadels) have tapered off, and half because a chunk of broker fees are now going to players in Citadels rather than being properly destroyed.  

Citadels remain a big topic in many market channels.  Specific recent highlights are things like a 0% fee Citadel dropping near Jita, and Hek not being in the citadel exclusion list.  Also, though there are some big bugs out there (Citadel timers are immune to TiDi), the strategic value of these structures is coming into focus.  We stand by the current advice that summer is the time to lock in positions.  There are a lot of set pieces that should make the fall very interesting when activity picks up.

The Rest Of The Summer

Coming into late July/August, things should be very quiet.  It's a heavy vacation season, and CCP has historically been very quiet in August.  General activity is staying well stoked given the traditional retraction, and it will be up to CCP to keep that fire stoked with activities.  Again, it's my strong advice that this is the time to grind up that cash, stash away those cheap PLEX, and generally recuperate ahead of the fall.  Many signals are pointing to an interesting season coming up and Alliance Tournament and EVE Vegas should get the hype trains moving.

Meanwhile, at ProsperHQ, we're still chugging away on our tool revamp.  Work starts in earnest this week on our data backend, and goals to get you all access to our data continues roughly on schedule.  Once a few more pieces come together, expect a "state of Prosper" blog in the next few weeks!