Tools
Записки программиста, обо всем и ни о чем. Но, наверное, больше профессионального.
2015-03-31
PayPal & Python
Махмуд, ведущий
разработчик в PayPal, рассказал нам, почему
мифы — такие мифы и как правильно
готовить Python в энтерпрайзе
I joined PayPal a
few years ago, and chose Python to work on internal applications, but
I’ve personally found production PayPal Python code from nearly 15
years ago.
Today, Python powers
over 50 projects, including:
Features and
products, such as eBay Now and RedLaser
Operations and
infrastructure, both OpenStack and proprietary
Mid-tier
services and applications, like the one used to set PayPal’s prices
and check customer feature eligibility
Monitoring
agents and interfaces, used for several deployment and security use
cases
Batch jobs for
data import, price adjustment, and more
And far too many
developer tools to count
In the coming series
of posts I’ll detail the initiatives and technologies that led the
eBay/PayPal Python community to grow from just under 25 engineers in
2011 to over 260 in 2014. For this introductory post, I’ll be
focusing on the 10 myths I’ve had to debunk the most in eBay and
PayPal’s enterprise environments
Myth #1: Python is a
new language
Myth #2: Python is
not compiled
Myth #3: Python is
not secure
Myth #4: Python is a
scripting language
Myth #5: Python is
weakly-typed
Myth #6: Python is
slow
Myth #7: Python does
not scale
Myth #8: Python
lacks good concurrency support
Myth #9: Python
programmers are scarce
Myth #10: Python is
not for big projects
Из этой десятки
мифов реальный интерес представляют
номера 3, 6, 7, 8.
Приводя примеры
из жизни, Махмуд на пальцах объясняет,
почему это всего лишь мифы а не горькая
правда жизни.
original post http://vasnake.blogspot.com/2015/03/paypa-python.html
Posted by Valentin at 02:32 0 comments
Labels: python
2015-03-30
Access for BigData
Airbnb
выпустил свой инструментарий для работы
с БигДата, что-то вроде MS Access, пользовательский
интерфейс к залежам данных в Hadoop.
Apartment-sharing
startup Airbnb has open sourced a tool called Airpal that the company
built to give more of its employees access to the data they need for
their jobs. Airpal is built atop the Presto SQL engine that Facebook
created in order to speed access to data stored in Hadoop.
Airbnb built Airpal
about a year ago so that employees across divisions and roles could
get fast access to data rather than having to wait for a data analyst
or data scientist to run a query for them. According to product
manager James Mayfield, it’s designed to make it easier for novices
to write SQL queries by giving them access to a visual interface,
previews of the data they’re accessing, and the ability to share
and reuse queries
A couple years ago,
Facebook created and then open sourced Presto as a means to solve
Hive’s speed problems. It still accesses data from Hive, but is
designed to deliver results at interactive speeds rather than in
minutes or, depending on the query, much longer. It also uses
standard ANSI SQL, which Kramolisch said is easier to learn than the
Hive Query Language and its “lots of hidden gotchas.”
Популярная
онлайн-площадка для краткосрочной
аренды жилья Airbnb представила свою новую
Open Source-разработку Airpal — инструмент для
анализа данных, основанный на движке
Presto SQL от Facebook.
До сих пор в
Airbnb использовалась система управления
данными Apache Hive, работающая на платформе
Hadoop и довольно сложная в плане использования
конечным пользователем. Новый же
инструмент — Airpal — имеет графический
интерфейс, позволяющий составлять
SQL-запросы к базам данных без особых
усилий. В его основе лежит PrestoDB — Open
Source-проект Facebook, запущенный пару лет
тому назад с целью решить проблему с
недостаточно высокой производительностью
Hive
original post http://vasnake.blogspot.com/2015/03/access-for-bigdata.html
Posted by Valentin at 02:23 0 comments
2015-03-26
Transfer learning и поиск похожих изображений
ML, transfer learning.
Использование
уже построенных моделей, наработанных
знаний, для решения других задач.
В качестве
аналогии приводится мой любимый пример,
когда мозг перестраивается и обучается
распознавать визуальные образы через
вкусовые или слуховые центры.
Авторы
проводят четкую границу между стандартным
машинным обучением и transfer learning. В
стандартном подходе у вас есть только
цель и набор данных, а задача заключается
в том, чтобы какими-либо методами достичь
этой цели. В рамках решения задачи вы
можете построить глубокую сеть,
предобучить ее жадным алгоритмом,
построить еще с десяток таких же и
каким-либо образом сделать ансамбль из
них. Но все это будет в рамках решения
какой-то одной проблемы, и время,
потраченное на решение такой проблемы,
будет сравнимо с суммарным временем,
потраченным на обучение каждой модели,
и ее предобучение.
А теперь
представьте, что есть две задачи, и
решали их, возможно, даже разные люди.
Один из них использует часть модели
другого (source task) для уменьшения временных
затрат на создание модели с нуля и
улучшения производительности своей
модели (target task). Процесс передачи знаний
от одной проблемы к другой и есть transfer
learning. А наш мозг, вероятно, так и поступает.
Как в примере выше, его реальная задача
почувствовать вкус рецепторами языка
и видеть глазами. Встает задача —
воспринимать визуальную информацию
рецепторами языка. И вместо того, чтобы
вырастить новые нейроны или же сбросить
веса старых и обучить их заново, мозг
просто слегка корректирует имеющуюся
нейронную сеть для достижения результата
Во второй части
статьи рассматривается практический
пример, в котором решалась задача поиска
похожих изображений.
При этом, на
первом этапе выделения свойств картинки,
была использована уже обученная сеть
победителей конкурса ImageNet.
При такой
вариативности датасета логично
предположить, что где-то в сети присутствует
эффективный извлекатель признаков, а
также классификатор, который и решает,
к какому классу относится изображение.
Хотелось бы достать этот самый извлекатель,
отделить его от классификатора и
поиспользовать его для обучения глубокого
автоенкодера из статьи о семантическом
хешировании
Очень интересно.
original post http://vasnake.blogspot.com/2015/03/transfer-learning.html
Posted by Valentin at 16:28 0 comments
Labels: ML
2015-03-24
ArcGIS Server Object Interceptors
ArcGIS SOI это новая
фишка для ArcGIS Server, прослойка, позволяющая
вмешиваться в запросы/ответы сервера.
They have the same
API as Server Object Extensions, and are intended to extend an ArcGIS
Server with custom capabilities. An SOI intercepts REST and/or SOAP
calls on a MapServer before and/or after it executes the operation on
an SOE or SO. Think servlet filters
Mansour сделал
showcase с использованием SOI и BigData (MemSQL), для
отображения на веб-карте маршрутов
такси
The good folks at
MemSQL bootstrapped a set of AWS instances with their “new”
engine and loaded the now-very-famous New York City taxis trips data.
This (very very small) set consists of about 170 million records with
geospatial and temporal information such as pickup and drop off
locations and times. Each trip has additional attributes such as
travel times, distances and number of passengers. It was up to me now
to query and display dynamically this information in a standard
WebMap on every map pan and zoom. What do I mean by “standard”
here, is that an out-of-the-box WebMap should be able to interact
with this MemSQL database without being augmented with a new layer
type or any other functionality. Thus the usage of an SOI. It will
intercept the call to an export image operation with a map extent as
an argument in a “stand-in” MapService and will execute a spatial
MemSQL call on the AWS instances. The result set is drawn on an
off-screen PNG image and is sent back to the requesting WebMap for
display as a layer on a map
original post http://vasnake.blogspot.com/2015/03/arcgis-server-object-interceptors.html
Posted by Valentin at 16:21 0 comments
2015-03-23
An army of frogs
Интересная
табличка нашлась на просторах Интернета.
Как для разных
животных и прочих насекомых называют
скопления, группы.
Ну, типа «стадо
коров».
we have received a
spate of questions on collective nouns or group names for all sorts
of critters. Perhaps the following, gleaned and compiled from several
sources, will help. I've tried to indicate proper usage for terms
that apply to a specific type of assemblage or to only a single
gender or age, but some of my sources were fairly general. Where my
sources disagreed on spelling, I let the majority rule. This listing
exhausts the meager pile of references available at Northern Prairie
Wildlife Research Center; if the animal that you are interested in
doesn't appear here, please, please, ask someone else, because I
can't help you!
Mammals
|
|
---|---|
Apes
|
A shrewdness
|
Asses
|
A pace
|
Badgers
|
A cete
|
Bats
|
A colony
|
Bears
|
A sloth, sleuth
|
Buffalo
|
A gang, an obstinacy (I suspect these refer to
old world buffalo; use "herd" for American bison)
|
Cats
|
A clowder, a pounce; for kittens...A kindle,
litter, an intrigue
|
Cattle
|
A drove, herd
|
Deer
|
A herd, bevy (refers only to roe deer)
|
Dogs
|
A litter (young), pack (wild), cowardice (of
curs); specific to hounds...A cry, mute, pack, kennel
|
Elephants
|
A herd
|
Elk
|
A gang
|
Ferrets
|
A business
|
Fox
|
A leash, skulk, earth
|
Giraffes
|
A tower
|
Goats
|
A tribe, trip
|
Gorillas
|
A band
|
Hippopotamuses
|
A bloat
|
Horses
|
A team, harras, rag (for colts), stud (a group
of horses belonging to a single owner, string (ponies)
|
Hyenas
|
A cackle
|
Kangaroos
|
A troop
|
Leopards
|
A leap
|
Lions
|
A pride
|
Martens
|
A richness
|
Moles
|
A labor
|
Monkeys
|
A troop, barrel
|
Mules
|
A pack, span, barren
|
Otters
|
A romp
|
Oxen
|
A team, yoke
|
Pigs
|
A drift, drove, litter (young), sounder (of
swine), team, passel (of hogs), singular (refers to a group of
boars)
|
Porcupines
|
A prickle
|
Rabbits
|
A colony, warren, nest, herd (domestic only),
litter (young); specific to hares...A down, husk
|
Rhinoceroses
|
A crash
|
Seals
|
A pod, herd
|
Sheep
|
A drove, flock, herd
|
Squirrels
|
A dray, scurry
|
Tigers
|
A streak
|
Whales
|
A pod, gam, herd
|
Wolves
|
A pack, rout or route (when in movement)
|
Birds
|
|
---|---|
Birds in general
|
A flight (in the air), flock (on the ground),
volary, brace (generally for gamebirds or waterfowl, referring to
a pair or couple killed by a hunter)
|
Bitterns
|
A sedge
|
Buzzards
|
A wake
|
Bobolinks
|
A chain
|
Chicks (of many species)
|
A brood; clutch
|
Coots
|
A cover
|
Cormorants
|
A gulp
|
Cranes
|
A sedge
|
Crows
|
A murder, horde
|
Dotterel
|
A trip
|
Doves
|
A dule, pitying (specific to turtle doves)
|
Ducks
|
A brace, flock (in flight), raft (on water)
team, paddling (on water), badling
|
Eagles
|
A convocation
|
Finches
|
A charm
|
Flamingos
|
A stand
|
Geese
|
A flock, gaggle (on the ground), skein (in
flight)
|
Grouse
|
A pack (in late season)
|
Gulls
|
A colony
|
Hawks
|
A cast, kettle (flying in large numbers), boil
(two or more spiraling in flight)
|
Herons
|
A sedge, a siege
|
Jays
|
A party, scold
|
Lapwings
|
A deceit
|
Larks
|
An exaltation
|
Mallards
|
A sord (in flight), brace
|
Magpies
|
A tiding, gulp, murder, charm
|
Nightingales
|
A watch
|
Owls
|
A parliament
|
Parrots
|
A company
|
Partridge
|
A covey
|
Peacocks
|
A muster, an ostentation
|
Penguins
|
A colony
|
Pheasant
|
A nest, nide (a brood), nye, bouquet
|
Plovers
|
A congregation, wing (in flight)
|
Ptarmigans
|
A covey
|
Rooks
|
A building
|
Quail
|
A bevy, covey
|
Ravens
|
An unkindness
|
Snipe
|
A walk, a wisp
|
Sparrows
|
A host
|
Starlings
|
A murmuration
|
Storks
|
A mustering
|
Swallows
|
A flight
|
Swans
|
A bevy, wedge (in flight)
|
Teal
|
A spring
|
Turkeys
|
A rafter, gang
|
Widgeons
|
A company
|
Woodcocks
|
A fall
|
Woodpeckers
|
A descent
|
Reptiles and Amphibians
|
|
---|---|
Crocodiles
|
A bask
|
Frogs
|
An army
|
Toads
|
A knot
|
Turtles
|
A bale, nest
|
Snakes, vipers
|
A nest
|
Fish
|
|
---|---|
Fish in general
|
A draft, nest, school, shoal (some authors
claim that the common "school" is a corruption of
shoal, and therefore incorrect)
|
Bass
|
A shoal
|
Herring
|
An army
|
Sharks
|
A shiver
|
Trout
|
A hover
|
Invertebrates
|
|
---|---|
Ants
|
A colony
|
Bees
|
A grist, hive, swarm
|
Caterpillars
|
An army
|
Clams
|
A bed
|
Cockroaches
|
An intrusion
|
Flies
|
A business
|
Gnats
|
A cloud, horde
|
Grasshoppers
|
A cloud
|
Hornets
|
A nest
|
Jellyfish
|
A smack
|
Locusts
|
A plague
|
Oysters
|
A bed
|
original post http://vasnake.blogspot.com/2015/03/an-army-of-frogs.html
Posted by Valentin at 16:39 0 comments
Labels: language
2015-03-22
Zope, ZODB и ACID
Памятка на
тему конфликтов чтения-записи и прочей
ACID-ности в БД Zope, ZODB.
ZODB provides
execution time consistency by raising read conflict errors. A simple
explanation of read conflict errors requires some extra details about
how ZODB works. ZODB transactions execute in parallel. When one
connection commits a transaction, it sends an invalidation message to
all the other connections; the invalidation messages lists all the
objects that were modified by the committed transaction. The next
time another connection commits a transaction, it will apply the
invalidation; i.e. it will ghostify all the objects in the
invalidation message.
If a connection is
asked to load an object from the database and it has received (but
not applied) an invalidation message for the object, it raises a
ReadConflictError. This error is necessary to avoid an inconsistent
read. When ZODB loads an object from storage, it always reads the
most recent revision. Since the connection has an invalidation
message for the object, the object was modified after the transaction
began. If it reads the current revision of that object, it could be
inconsistent with other objects read before the transaction began.
The way ZODB and
threading works is that each thread that uses the database gets its
own connection to the database. Each connection gets its own copy of
your object. All of the threads can read and change any of the
objects. ZODB keeps all of these objects synchronized between the
threads. The upshot is that you don’t have to do any locking or
thread synchronization yourself. Your code can act as though it is
single threaded.
However,
synchronization problems can occur when objects are changed by two
different threads at the same time.
Imagine that thread
1 gets its own copy of object A, as does thread 2. If thread 1
changes its copy of A, then thread 2 will not see those changes until
thread 1 commits them. In cases where lots of objects are changing,
this can cause thread 1 and 2 to try and commit changes to object 1
at the same time.
When this happens,
ZODB lets one transaction do the commit (it “wins”) and raises a
‘ConflictError’ in the other thread (which “looses”).
def writing_to_zdb(): zdb = open_db() retry = 0 while retry < max_retry: try: # this is where you write to, delete from, & change the DB # for example, this creates a new {} that is persistent zdb['root']['new_dict'] = PersistentDict() transaction.get().commit() except ConflictError: retry += 1 time.sleep(sleep_delay) pass else: break else: # retried too many times close_db(zdb) return "Error - transaction could not complete" close_db(zdb) return "Data written successfully" |
Еще?
original post http://vasnake.blogspot.com/2015/03/zope-zodb-acid.html
Posted by Valentin at 19:01 0 comments
Labels: Zope(Plone)
Perfect Continuous
Язык и время.
Настоящее,
прошедшее и будущее.
К каждому из
трех времён четыре варианта действий:
простое
(неопределенное), длительное, совершённое,
длительное совершённое.
Непонятно?
Смотрите
картинку:
original post http://vasnake.blogspot.com/2015/03/perfect-continuous.html
Posted by Valentin at 18:18 1 comments
2015-03-20
Webogram
Я тут недавно
начал пользоваться IM Telegram. Это тот,
который спонсируется Павлом Дуровым.
Мне очень
нравится. Рекламы нет, работает быстро,
ресурсов жрет немного, все сообщения
появляются сразу на всех залогиненных
устройствах, картинки и файлы отсылаются
на раз-два.
Не нравится
только одно – жесткая привязка к номеру
телефона. Я предпочитаю старый добрый
способ – через электропочту.
Говорят –
приватный, безопасный. Не знаю, мне
пофиг. Я не жду от них особой защещенности
моей переписки или моих данных.
Зато есть
веб-клиент и он с открытым кодом.
Добротно
написанная на Ангуляре веб-аппликуха,
учитесь фронтэндщики.
original post http://vasnake.blogspot.com/2015/03/webogram.html
Posted by Valentin at 18:13 0 comments
Labels: im
Bulk Synchronous Parallel
Движуха
под лейблом Big Data прет бульдозером, не
остановишь. И это круто.
движок для
высокопроизводительных BSP-вычислений
Apache Hama теперь поддерживает не только
Apache Mesos, но и Hadoop YARN
Apache Hama is a
High-Performance BSP computing engine, which can be used to perform
compute-intensive general scientific BSP applications, Google’s
Pregel-like graph applications, and machine learning algorithms.
YARN is the resource
management technology that lets multiple computing frameworks run on
the same Hadoop cluster using the same underlying storage. So, for
example, a company could analyze the data using MapReduce, Spark, and
Apache Hama.
“From the next
release, you’ll be able to submit scientific BSP applications to
the existing open source Hadoop, CDH, and HDP clusters without any
installation” said Edward J. Yoon(@eddieyoon), a original creator
of Apache Hama
This contribution is
mainly coming from Samsung Electronics. “Unlike most web services
companies, our challenge is numerical or signal data, not text data.
That’s why we’re investing in High-Performance computing for
scientific advanced analytics.” said SeungHun Jeon, a Head of Cloud
Tech Lab at Samsung Electronics.
original post http://vasnake.blogspot.com/2015/03/bulk-synchronous-parallel.html
Posted by Valentin at 00:48 0 comments
Подписаться на:
Сообщения (Atom)
Архив блога
-
▼
2015
(120)
-
▼
марта
(28)
- PayPal & Python
- Access for BigData
- Transfer learning и поиск похожих изображений
- ArcGIS Server Object Interceptors
- An army of frogs
- Zope, ZODB и ACID
- Perfect Continuous
- Webogram
- Bulk Synchronous Parallel
- Задачка
- Балансируем
- PHP → Python
- Peaceful Warrior
- Artificial Neurons and Single-Layer Neural Networks
- Naudoc/Zope 2.10 Skins
- SSL/TLS
- Ceph
- Четкость формулировок
- Customizing Bootstrap
- MongoDB 3.0
- Прогород
- Две стрелы. Детектив каменного века
- Ceph
- ML и предсказание ситуации на дорогах
- Mixin
- Naudoc/Zope 2.10.7 Products
- AdvoCam FD Black-GPS
- Yes Man
-
▼
марта
(28)
Ярлыки
linux
(241)
python
(191)
citation
(186)
web-develop
(170)
gov.ru
(159)
video
(124)
бытовуха
(115)
sysadm
(100)
GIS
(97)
Zope(Plone)
(88)
бурчалки
(84)
Book
(83)
programming
(82)
грабли
(77)
Fun
(76)
development
(73)
windsurfing
(72)
Microsoft
(64)
hiload
(62)
internet provider
(57)
opensource
(57)
security
(57)
опыт
(55)
movie
(52)
Wisdom
(51)
ML
(47)
driving
(45)
hardware
(45)
language
(45)
money
(42)
JS
(41)
curse
(40)
bigdata
(39)
DBMS
(38)
ArcGIS
(34)
history
(31)
PDA
(30)
howto
(30)
holyday
(29)
Google
(27)
Oracle
(27)
tourism
(27)
virtbox
(27)
health
(26)
vacation
(24)
AI
(23)
Autodesk
(23)
SQL
(23)
Java
(22)
humor
(22)
knowledge
(22)
translate
(20)
CSS
(19)
cheatsheet
(19)
hack
(19)
Apache
(16)
Manager
(15)
web-browser
(15)
Никонов
(15)
functional programming
(14)
happiness
(14)
music
(14)
todo
(14)
PHP
(13)
course
(13)
scala
(13)
weapon
(13)
HTTP. Apache
(12)
Klaipeda
(12)
SSH
(12)
frameworks
(12)
hero
(12)
im
(12)
settings
(12)
HTML
(11)
SciTE
(11)
USA
(11)
crypto
(11)
game
(11)
map
(11)
HTTPD
(9)
ODF
(9)
купи/продай
(9)
Photo
(8)
benchmark
(8)
documentation
(8)
3D
(7)
CS
(7)
DNS
(7)
NoSQL
(7)
cloud
(7)
django
(7)
gun
(7)
matroska
(7)
telephony
(7)
Microsoft Office
(6)
VCS
(6)
bluetooth
(6)
pidgin
(6)
proxy
(6)
Donald Knuth
(5)
ETL
(5)
NVIDIA
(5)
Palanga
(5)
REST
(5)
bash
(5)
flash
(5)
keyboard
(5)
price
(5)
samba
(5)
CGI
(4)
LISP
(4)
RoR
(4)
cache
(4)
car
(4)
display
(4)
holywar
(4)
nginx
(4)
pistol
(4)
spark
(4)
xml
(4)
Лебедев
(4)
IDE
(3)
IE8
(3)
J2EE
(3)
NTFS
(3)
RDP
(3)
holiday
(3)
mount
(3)
Гоблин
(3)
кухня
(3)
урюк
(3)
AMQP
(2)
ERP
(2)
IE7
(2)
NAS
(2)
Naudoc
(2)
PDF
(2)
address
(2)
air
(2)
british
(2)
coffee
(2)
fitness
(2)
font
(2)
ftp
(2)
fuckup
(2)
messaging
(2)
notify
(2)
sharepoint
(2)
ssl/tls
(2)
stardict
(2)
tests
(2)
tunnel
(2)
udev
(2)
APT
(1)
CRUD
(1)
Canyonlands
(1)
Cyprus
(1)
DVDShrink
(1)
Jabber
(1)
K9Copy
(1)
Matlab
(1)
Portugal
(1)
VBA
(1)
WD My Book
(1)
autoit
(1)
bike
(1)
cannabis
(1)
chat
(1)
concurrent
(1)
dbf
(1)
ext4
(1)
idioten
(1)
join
(1)
krusader
(1)
license
(1)
life
(1)
migration
(1)
mindmap
(1)
navitel
(1)
pneumatic weapon
(1)
quiz
(1)
regexp
(1)
robot
(1)
science
(1)
serialization
(1)
spatial
(1)
tie
(1)
vim
(1)
Науру
(1)
крысы
(1)
налоги
(1)
пианино
(1)