How to generalize (algorithmically)

Posted on September 18, 2018 by Sebastien Bubeck

A couple of months ago I taught an introduction to statistical learning theory. I took inspiration from two very good introductory books on SLT: “Foundations of ML”, and “Understanding Machine Learning: From Theory to Algorithms”. I also covered some classical results about nearest neighbors from the seminal book “A Probabilistic Theory of Pattern Recognition”. The lectures were recorded and I put the videos on the youtube channel. Sadly I still did not learn the basic lesson to stick to a black pen when teaching on a whiteboard, so at (many) times it is hard to read what I write. I also got sick after the first lecture so my voice is quite weak in the following lectures, making it hard to hear too… Despite these fatal flaws some people still found the material useful, so check it out if you are interested. Below I give a rough syllabus for the lectures, and I also give some notes on “algorithmic generalization” which was in my opinion the highlight of the lectures (you can also check my earlier notes on SLT here and here).

Syllabus

Lecture 1: Introduction to the statistical learning theory framework, its basic question (sample complexity) and its canonical settings (linear classification, linear regression, logistic regression, SVM, neural networks). Two basic methods for learning: (i) Empirical Risk Minimization, (ii) Nearest neighbor classification.

Lecture 2: Uniform law of large numbers approach to control the sample complexity of ERM (includes a brief reminder of concentration inequalities). Application: analysis of bounded regression (includes the non-standard topic of type/cotype and how it relates to different regularizations such as in LASSO).

Lecture 3: Reminder of the first two lectures and relation with the famous VC dimension. How to generalize beyond uniform law of large numbers: stability and robustness approaches (see below).

Lecture 4: How to generalize beyond uniform law of large numbers: information theoretic perspective (see below), PAC-Bayes, and online learning. Brief discussion of margin theory, and an introduction to modern questions in robust machine learning.

 

Some notes on algorithmic generalization

Let \mathcal{X}, \mathcal{Y} be input/output spaces. Let \ell : \mathcal{Y} \times \mathcal{Y} \rightarrow [0,1] be a loss function, \mu a probability measure supported on \mathcal{X} \times \mathcal{Y}, and \underline{h} : (\mathcal{X} \times \mathcal{Y})^m \rightarrow \mathcal{Y}^\mathcal{X} a learning rule (in words \underline{h} takes as input a dataset of m examples, and output a mapping from \mathcal{X}-inputs to \mathcal{Y}-outputs). With a slight abuse of notation, for z=(x,y) \in \mathcal{X} \times \mathcal{Y} and h \in \mathcal{Y}^{\mathcal{X}}, we write \ell(h, z) := \ell(h(x), y). We define the generalization of \underline{h} on \mu by:

    \[ \mathrm{gen}(\underline{h} , \mu) = \mathbb{E}_{S=(z_1,\hdots,z_m) \sim \mu^{\otimes m}, z \sim \mu} \left[ \ell(\underline{h}(S), z) - \frac1{m} \sum_{i=1}^m \ell(\underline{h}(S), z_i) \right] \,. \]

In words, if \mathrm{gen}(\underline{h} , \mu) \leq \epsilon then we expect the empirical performance of the learned classifier \underline{h}(S) to be representative of its performance on a fresh out-of-sample data point, up to an additive \epsilon. The whole difficulty of course is that the empirical evaluation is done with the same dataset S that is used for training, leading to non-trivial dependencies. We should also note that in many situations one might be interested in the two-sided version of the generalization, as well as high probability bounds instead of bounds in expectation. For simplicity we focus on \mathrm{gen}(\underline{h} , \mu) here.

The most classical approach to controlling generalization, which we covered in details in previous notes, is via uniform law of large numbers. More precisely assuming that the range of the learning rule is some hypothesis class \mathcal{H} one trivially has

    \[ \mathrm{gen}(\underline{h} , \mu) \leq \mathbb{E}_{S, z} \,\, \sup_{h \in \mathcal{H}} \left[ \frac1{m} \sum_{i=1}^m \ell(h, z_i) - \ell(h, z) \right] \,. \]

However this approach might be too coarse when the learning rule is searching through a potentially huge space of hypothesis (such as in the case of neural networks). Certainly such uniform bound has no chance of explaining why neural networks with billions of parameters would generalize with a data set of merely millions of examples. For this one has to use algorithm-based arguments.

Stability

The classical example of algorithmic generalization is due to Bousquet and Elisseeff 2002. It is a simple rewriting of the generalization as a stability notion:

    \begin{eqnarray*} \mathrm{gen}(\underline{h} , \mu) & = & \mathbb{E}_{S, z, i \sim [n]} \left[\ell(\underline{h}(S), z) - \ell(\underline{h}(S), z_i) \right] \\ & = & \mathbb{E}_{S, z, i \sim [n]} \left[\ell(\underline{h}(S^{-i}), z_i) - \ell(\underline{h}(S), z_i) \right] \,, \end{eqnarray*}

where S^{-i}=(z_1,\hdots, z_{i-1}, z, z_{i+1}, \hdots, z_m). This viewpoint can be quite enlightening. For example in the uniform law of large numbers view, regularization enforces small capacity, while in the stability view we see that regularization ensures that the output hypothesis is not too brittle (this was covered in some details in the previous notes).

Robustness

The next approach I would like to discuss is related to deep questions about current machine learning methods. One of the outstanding problem in machine learning is that current algorithms are not robust to even mild shift of distribution at test time. Intuitively this lack of robustness seem to indicate a lack of generalization. Can we formalize this intuition? I will now give one such formal link between robustness and generalization due to Xu and Mannor 2010, which shows the reverse direction (robustness implies generalization). At some level robustness can be viewed as a “stability at test time” (while in Bousquet and Elisseeff we care about “stability at training time”).

Xu and Mannor define (K, \epsilon(\cdot))-robustness as follows: assume that \mathcal{Z} can be partitioned into K sets \{C_k\}_{k \in [K]} such that if z and z_i are in the same set C_k then

    \[ | \ell(\underline{h}(S), z_i) - \ell(\underline{h}(S), z) | \leq \epsilon(S) \,. \]

A good example to have in mind would be a binary classifier with large margin, in which case K corresponds to the covering number of \mathcal{X} at the scale given by the margin. Another (related) example would be regression with a Lipschitz function. In both cases K would be typically exponential in the dimension of \mathcal{X}. The key result of Xu and Mannor that we prove next is a generalization bound of order \sqrt{K / m}. In any situation of interest this seems to me to be a pretty weak bound, yet on the other hand I find the framework to be very pretty and it is of topical interest. I would be surprised if this was the end of the road in the space of “generalization and robustness”.

Theorem (Xu and Mannor 2010):

A (K, \epsilon(\cdot))-robust learning rule satisfies

    \[ \mathrm{gen}(\underline{h} , \mu) \leq \mathbb{E}_{S} \, \epsilon(S) + \sqrt{\frac{K}{m}} \,. \]

Proof: Let N_k = |S \cap C_k| and note that \mathbb{E}_S \, N_k = m \mu(C_k). Now one has for a robust h:

    \begin{align*} \mathbb{E}_z \ \ell(h, z) & =\sum_{k=1}^K \mu(C_k) \mathbb{E}_z [\ell(h,z) | z \in C_k] \\ & = \sum_{k=1}^K \frac{N_k}{m} \mathbb{E}_z [\ell(h,z) | z \in C_k] + \sum_{k=1}^K \left(\mu(C_k) - \frac{N_k}{m}\right) \mathbb{E}_z [\ell(h,z) | z \in C_k] \\ & \leq \epsilon(S) + \frac{1}{m} \sum_{i = 1}^m \ell(h, z_i) + \sum_{k=1}^K \left|\mu(C_k) - \frac{N_k}{m}\right| \, . \end{align*}

It only remains to observe that

    \[ \mathbb{E}_S \, \sum_{k=1}^K \left|\mu(C_k) - \frac{N_k}{m}\right| \leq \frac{1}{m} \sum_{k=1}^K \sqrt{\mu(C_k)} \leq \sqrt{\frac{K}{m}} \, . \]

 

Information theoretic perspective

Why do we think that a lack of robustness indicate a lack of generalization? Well it seems to me that a basic issue could simply be that the dataset was memorized by the neural network (which be a very non-robust way to learn). If true then one could basically find all the information about the data in the weights of the neural network. Again, can we prove at least the opposite direction, that is if the output hypothesis does not retain much information from the dataset then it must generalize. This is exactly what Russo and Zou 2016, where they use the mutual information \mathrm{I}(\underline{h}(S), S) as a measure of the “information” retained by the trained hypothesis about the dataset. More precisely they show the following result:

Theorem (Russo and Zou 2016):

    \[ \mathrm{gen}(\underline{h} , \mu) \leq \sqrt{\frac{\mathrm{I}(\underline{h}(S), S)}{2 m}} \, . \]

Note that here we have assumed that the codomain of the learning rule \underline{h} consists of deterministic maps from inputs to outputs, in which case the mutual information \mathrm{I}(\underline{h}(S), S) is simply the entropy \mathrm{H}(\underline{h}(S)). However the proof below also applies to the case where the codomain of the learning rule consists of probability measures, see e.g., Xu and Raginsky 2017. Let us now conclude this (long) post with the proof of the above theorem.

The key point is very simple: one can view generalization as a decoupling property by writing:

    \[ \mathrm{gen}(\underline{h} , \mu) = \mathbb{E}_{S, S' \sim \mu^{\otimes m}} [ F(\underline{h}(S), S') - F(\underline{h}(S), S) ] \,, \]

where F(h, S) = \frac{1}{m} \sum_{z \in S} \ell(h, z).

Now the theorem follows straightforwardly (if one knows Hoeffding’s lemma) from an application of the following beautiful lemma:

Lemma:

Let f: \mathcal{X} \times \mathcal{Y} \rightarrow \mathbb{R}. Let (X,Y) be random variables in \mathcal{X} \times \mathcal{Y} and \bar{X}, \bar{Y} be mutually independent copies of X and Y. Assume that f(\bar{X}, \bar{Y}) is \sigma-subgaussian (i.e., \log \mathbb{E} \exp( \lambda (f(\bar{X}, \bar{Y}) - \mathbb{E} f) ) \leq \lambda^2 \sigma^2 /2, \forall \lambda) then

    \[ \mathbb{E}[ f(X,Y) - f(\bar{X}, \bar{Y}) ] \leq \sqrt{2 \sigma^2 \mathrm{I}(X, Y)} \]

Proof: The mutual information is equal to the relative entropy between the distribution of (X,Y) and the distribution of (\bar{X}, \bar{Y}). Recall also the variational representation of the relative entropy which is that the map \mu \mapsto \mathrm{Ent}(\mu, \nu) is the convex conjugate of the log-partition function \Phi_{\nu}(g) := \log(\int \exp(g(x)) d \nu(x)). In particular one has a lower bound on the mutual information for any such g which means:

    \[ \mathrm{I}(X, Y) \geq \mathbb{E}[ g(X,Y) ] - \log \mathbb{E} [ \exp(g(\bar{X},\bar{Y})) ] \,. \]

Now it only remains to use the definition of subgaussianity, that is take g = \lambda f, and optimize over \lambda.

This entry was posted in Machine learning. Bookmark the permalink.

44 Responses to "How to generalize (algorithmically)"

  • By Medya Kazanı May 10, 2021 - 8:31 am

    takipçi instagram takipçi satın al medya kazani ndan sınırsız takipçi satın alabilirsiniz.

    Reply
  • By jake May 9, 2021 - 6:35 pm

    Rahat depolama, bizi arayarak eşya depolama hizmetimizden yararlanabilir, eşyalarınızın durumunu sorgulayabilir ayrıca en uygun fiyat teklifini alabilirsiniz.

    depo kiralama

    Reply
  • By evcil hayvan ürünleri May 5, 2021 - 11:55 am

    evcil hayvan ürünleri indirimli bir şekilde satın almak için siteye ulaşmanız yeterli. Ben çok memnun kaldım özellikle kargo teslimatı çok hızlıydı.

    Reply
    • By Ronan Smith May 7, 2021 - 12:47 am

      McAfee offer several solutions such as McAfee Login, McAfee Security Login, and McAfee Login Portal, where you can access your McAfee Login My Account Page and protect your email address, personal data, and any mobile theft or loss. Visit McAfee Login website to access your McAfee Login Account Page at any time.

  • By gel bôi trơn có mùi May 4, 2021 - 6:10 am

    follow my website

    Reply
  • By Ahmet Dilber May 3, 2021 - 7:22 am

    As Ahmet Dilber Medical Center, we have been serving in the field of plastic surgery in Istanbul Metroport Busidence since March 2020. We mainly provide services in the field of Nose Aesthetics, Breast Aesthetics and Bichectomy. Body areas such as Liposuction, Eyelid, Eyebrow Lift. The sterile environment and advanced equipment in our operating rooms combined with our experienced healthcare professionals provided us with a 99% satisfaction rate in more than 150 plastic surgery applications. Under the leadership of Aesthetic, Plastic and Reconstructive Surgery Specialist Ahmet Dilber, we carry out our procedures in special rooms to make our customers feel more comfortable and satisfied.

    Reply
  • By Ayrıldığım Kocamla Barışma Duası May 1, 2021 - 5:58 pm

    Ayrıldığım Kocamla Barışma Duası

    #Soğutmaduası#Bağlamaduası#Gerigetirmebüyüsü#Aşkbüyüsü#Bağlamabüyüsü

    https://www.muharremhocam.com/ayrildigim-kocamla-barisma-duasi/

    Reply
  • By Regülatörrr April 27, 2021 - 4:44 pm

    REGÜLATÖR
    1975 yılında kurulmuş olan GÜVEN-İŞ; Servo Voltaj Regülatörü, Transformatör ve elektrik endüstrisi için Akü kabini, Battery bank, Elektrik panosu, Rack kabin, Regülatör şasesi, Metal kutu ve muhtelif Kabin ve şaseler üretmekte olup 40 yılı aşkın süredir bilgi birikimi ve tecrübesi ile sektörünün öncü ve köklü kuruluşlarından biri haline gelmiştir.

    ISO 9001 kalite yönetim sistemi altında üretim yapan GÜVEN-İŞ tüm ürünlerini uzman personeli, modern makinaları ve kaliteden taviz vermeyen çalışma sistemiyle ilgili standartlar kapsamında CE belgeli olarak üretmektedir.

    Standart üretim yanında GÜVEN-İŞ, Kullanıcı talebine göre özel ürünlerin geliştirilmesi ve imalatı konusunda oldukça esnek olabilir. Bütün bunlar, uzun yıllar süren uygulanmış teknolojik gelişmeyle elde edilen deneyim sayesinde sağlanmaktadır. Bu tür gelişmeler, teknik personelin her bir “özel ürün” için elektrikli ve mekanik tasarımları hızlı ve uygun maliyetli bir şekilde hazırlanmasını sağlayan bilgi ve yeterliliğe sahiptir.

    GÜVEN-İŞ her geçen gün üretim kapasitesini artırmak amacıyla yatırımlar yapmaya devam etmektedir.

    Reply
  • By Regülatörr April 27, 2021 - 4:43 pm

    Güvenİş Regülatör Çeşitleri

    Güvenİş regülatör denilince akla gelen ilk isim olması ile bilinir. Kurumsal ve bireysel kullanım için regülatörü üreten firma, geniş ürün yelpazesi ile faz ve voltaj uyum sorunu yaşamadan kendinize uygun olan regülatörünüzü bulmanızı sağlıyor. Kullanım için yeterliliğe göre DVR ve SVR serisi ve alt modellerini seçebilirsiniz. Statik ve mikroişlemcili regülatör ile piyasanın en iyisi olmaya devam ediyor.

    Statik ve mikroişlemcili regülatörler aşırı gerilim için kullanılmaz. Akım ayarlamasını çıkış fazına göre kendi ayarlamaktadır. Bu regülatörler akımın trafo yakınlığı nedeniyle yüksek veya trafo uzaklığı nedeni ile az gelmesi gibi durumlar da akımı lokal olarak ayarlar. Dalgalı akım gelmesi durumunda kullanılmaktadır.

    Reply
  • By Regülatör April 27, 2021 - 4:42 pm

    Regülatör Nedir?

    Elektrik enerjisinin fazla kullanıldığı alanlarda özellikle fabrika, sanayi gibi yerlerde voltaj ayarını yapmak için halk dilinde yükselteç olarak bilinen bir sistem vardır. Buna regülatör denir. Regülatör, enerjinin kademeli olarak ayarlanmasını sağlar. Piyasa da regülatör satın alabileceğiniz birçok adres vardır. Bunlardan biri de Güven- İş Elektrik Elektronik San. Ve Tic. LTD ŞTİ (www.guvenisregulator.com.tr) firmasıdır.

    Reply
  • By DS Home Plus April 27, 2021 - 8:20 am

    We bring quality to the forefront with our Havana 8 piece dining set, Cherry Brown bedroom set, as well as trying to increase your comfort with homeplus furniture sofas and many other excellent furniture.

    Reply
  • By Nap Queens April 26, 2021 - 7:10 am

    You will maximize your comfort with Nap Queen blankets, nap beds and nap mattresses.

    Reply
  • By tüp bebek ve doğum April 25, 2021 - 5:11 pm

    Prof. Dr. Suat Dede halen Acıbadem Üniversitesi Tıp Fakültesi Kadın Hastalıkları ve Doğum Anabilim Dalında öğretim üyesi olarak görev yapmaktadır. Ayrıca üniversitenin afiliye hastanesi olan Acıbadem Maslak Hastanesinde de Kadın Hastalıkları ve Doğum uzmanı olarak çalışmaktadır.

    Reply
  • By Germany iptv April 25, 2021 - 5:57 am

    Almanya IPTV ile donmadan sorunsuz kaliteli full HD IPTV izle

    Reply
  • By Bez Çanta April 24, 2021 - 8:31 am

    Günümüzün sağlıklı ve doğal ürünleri arasında ilk sıralarda yer alan bez çantalar, reklam maksadıyla kullanıldığı gibi kadınların da sıklıkla tercih ettiği ürünler arasındadır. Ham bez çanta ürünler, tamamen doğal ve % 100 pamuk yapısı ile gönül rahatlığı ile kullanılabilir. Müşterilerimizin istediği ebatta ve baskıda üretilen bez çanta ürünleri promosyon maksadıyla da kullanabilirsiniz. Tercih edeceğiniz ürün yapısına ve sipariş sayısına göre ürün fiyatları değişmektedir.
    https://www.ucuzbezcanta.com/ internet sitemizi ziyaret ederek birbirinden kaliteli ve birbirinden güzel bez çantaları inceleyebilirsiniz. Web sitemizde yer alan formu doldurarak ürünlerle ilgili size uzman ekibimizin ulaşmasını sağlayabilirsiniz. Ekibimiz kısa sürede size ulaşacak ve gerekli bilgilendirmeyi yapacaktır. Sipariş verdiğiniz ürünler kısa sürede elinize ulaşacaktır.

    Reply
  • By Voguish Furniture April 24, 2021 - 6:50 am

    As Voguish Furniture, together with our partners around the world, we present the most beautiful and precious furniture to your liking in our furniture store on Fondren in Houston, TX.

    Reply
  • By Nolan Meyer April 23, 2021 - 8:19 am

    If you write about these algorithms on instagram you should know how to get likes for such posts. Most of people won’t understand it and won’t like. Read this article https://getblogo.com/instagram-likes-the-way-to-show-that-your-profile-is-as-successful-as-you-want-it-to-be/ about insta likes and where to get it

    Reply
  • By Maksat İstihdam April 23, 2021 - 2:38 am

    Web sitemizi takip ederek ülkemizdeki en güncel gelişmelerden anında haberdar olabilirsiniz. Sitemizde güncel iş haberlerinden, politik gelişmelere, ekonomi haberlerinden spor haberlerine kadar birçok kategoride haber başlığı bulabilirsiniz. Maksat İstihdam olarak internet kullanıcılarının en kaliteli ve güncel haber başlıklarına ulaşmasını sağlıyoruz.
    Herkesin mutlaka ilgili olduğu bir haber başlığı vardır. Herkes ilgili olduğu alanla ilgili güncel gelişmeleri yakından takip etmek ister. Bunun için de her web sitesinde yayınlanan haber başlıklarına itibar etmek mümkün değildir. Bu kapsamda web sitemize güvenebilir, en güncel ve doğru haberleri sitemizden öğrenebilirsiniz. https://www.maksatistihdam.com/ web sitemizi ziyaret ederek ilgili olduğunuz alanla ilgili güncel gelişmeleri takip edebilirsiniz. En doğru ve güncel haberler Maksat İstihdam’da sizleri bekliyor.

    Reply
  • By canli bahis forum siteleri April 22, 2021 - 6:15 am

    deneme bonusu veren siteler 2021 yılında en güncel bonus veren siteler çevrimsiz yatırımsız bonuslar ve yatırım şartsız bonus veren sitelerin bulunduğu canli bahis ve casino forum sitesi bonus piyasasi adresinde en güvenilir canli bahis ve casino sitelerinin sponsorluğunda sizde türkiyenin en güvenilir bahis forum sitesine üye olabilirsiniz . risksiz bahis veren siteler hoş geldin bonusu veren siteler çevrimsiz yatırım bonusları ve daha fazlasını bulabileceğiniz canli bahis forumunda canli casino para nasıl kazanılır slot oyunlarından nasıl para kazanılır canli bahis para kazanma gibi konuları paylaşım yapabilir veya yapılan paylaşımlara yorum atabilirsiniz. türkiyenin iddaa forum sitesi bonus piyasasi maç öncesi bahis kuponları canli bahis kuponları kombine kuponlar sistem bahisi canli bahis analizi gibi konuları iddaa tahmin alanımızda bulabileceksiniz. deneme bonusu veren siteler 2021 casino deneme bonusu 50 tl deneme bonusu 100 tl deneme bonusu gibi tüm konular sitemizde mevcuttur. bahis siteleri güncel giriş adresleri güvenilir bahis casino sitelerinin güncel giriş adresleri bedava bonus veren siteler foruma özel bedava bonus ve etkinlik ödülleri bu adreste. telegram bonusu veren siteler telegram sohbet kanalları bedava bonus telegram whatsapp bonusu veren siteler doğum günü bonusu veren siteler ilk kayıt bonusu veren siteler yeni üyelere bonus herşey bonus piyasasi canli bahis ve casino forum sitesinde.

    https://www.bonuspiyasa.com/

    Reply
  • By Medya Kazanı April 17, 2021 - 7:10 am

    İt web site very nice

    Reply
  • By Takipçi Al April 17, 2021 - 7:08 am

    great job

    Reply
  • By Andrew April 16, 2021 - 2:06 am

    Thanks

    Reply
  • By haberr.org April 15, 2021 - 2:38 pm

    Haberler,Son Dakika Haberleri,Haber,Teknoloji

    Haberler,Son Dakika Haberleri,Haber,Teknoloji

    Reply
  • By sağlık güzellik yaşam April 12, 2021 - 1:06 pm

    Sağlık hakkında bilgiler, Gıda takviyeleri , Güzellik cilt bakımı ve Ev & Dekorasyon hakkında herşey Sağlık Güzellik Cilt Bakımı bulabilirsiniz.

    Reply
  • By hulu accounts April 12, 2021 - 12:52 am

    great story and great blog

    Reply
  • By antalya havalimanı transfer April 10, 2021 - 4:40 pm

    antalya havalimanı transfer great post

    Reply
  • By Deniz April 10, 2021 - 9:40 am

    Deniz motoru pervanesi, motordan aldığı güç ile dönü kuvveti kazanır ve elde ettiği dönü ile su taşıtının hareket etmesini sağlar. Bu nedenle söz konusu malzeme, deniz taşıtları için son derece önemli ekipmanlardır. Deniz motoru pervanesi çeşitlerini web sitemizden inceleyerek kolayca satın alabilirsiniz. Web sitemizdeki ürünleri incelediğinizde her türlü deniz motoru ile ilgili pervane çeşitlerinin temin edilebildiğini göreceksiniz.
    Deniz motoru pervanesinin görevi ve öneminin bilincinde olan kişiler, en kaliteli ürünü cazip fiyata temin etmek isterler. https://www.sanalmarine.com web sitemizdeki ürünleri incelediğinizde tüm ürünlerin üstün kalitede ve cazip fiyatla satışa sunulduğunu göreceksiniz. Siz de kaliteyi uygun fiyata satın almak istiyorsanız web sitemizdeki ürünleri inceleyebilirsiniz.

    Reply
  • By American Siding April 9, 2021 - 2:50 am

    American Siding ile dayanıklı bir dış cephe kaplaması yaptırabilirsiniz. Firmamız bünyesinde yaptıracağınız dış cephe kaplama uygulaması ile bina yalıtımında % 55’e varan tasarruf sağlayacaksınız. American Siding, tam 50 yıl hizmet garantisi vererek dış cephe kaplama işlerini en iyi şekilde yapmaktadır. Üstelik alacağınız kaliteli hizmet sonrasında da müşteri desteğimizi devam ettiriyoruz.
    Firmamız bünyesinde yapılan tüm dış cephe kaplama işlerinde sağlam ve dayanıklı malzemeler kullanılmaktadır. Yapılan uygulama sonrasında arzu ettiğiniz tasarrufun elde edildiğini göreceksiniz. Sağlıklı şekilde kullanacağınız ürünlerimiz uzun ömürlü yapısı ile kullanıcıyı tatmin etmektedir. https://www.sidingturk.com/ web sitemizden yapılan faaliyetler ve dış cephe kaplama özellikleri ile ilgili ayrıntılı bilgi alabilirsiniz. Amerikan Siding ile estetik ve modern bir görünüme hazır olun!

    Reply
  • By StarCaliper April 8, 2021 - 3:33 am

    StarCaliper | Brake Caliper Repair Kit
    We are manufacturing Brake Caliper Repair Kits for European heavy vehicles. Our repair kits are for calipers as: KNORR, MERITOR, WABCO, HALDEX, BPW, SAF, Z-CAM, BREMBO, HYDRAULIC. We also manufature: CALIPER CARRIERS, SENSOR CABLES, CABLE DEVICES, CALIPER CONNECTING BOLTS and CALIPER REPAIR TOOLS.
    https://www.starcaliper.com

    Reply
  • By Free Account GO April 3, 2021 - 9:35 pm

    Different games that emerged with the developments in the digital game industry are followed by many users around the world. With the spread of survival-themed games, interesting productions have various features.

    The game named Garena Free Fire, which is easily preferred in smart phone operating systems such as Android and iOS, has been preferred by many people recently. Players do a lot of research about free fire accounts. It is possible to play the game comfortably with free accounts. We continue to share special free fire account free on our website!

    Reply
  • By Snapper News March 31, 2021 - 7:52 am

    Nice post, it was very interesting and useful.
    Also checkout my article by clicking the link below
    f95zone

    Reply
  • By mason March 29, 2021 - 7:42 pm

    Nice Info! Visit https://antivirus-setup.co/ for Best antivirus sofware of 2021

    Reply
  • By Köpek eğitim çiftliği March 26, 2021 - 7:30 am

    Köpek eğitim çiftliklerinde profesyonel hayvan sever eğitimciler tarafından oyunlarla ödüllerle her yönden eğitim verilmektedir. Örnek olarak;tuvalet sıkıntısı, havlama, ısırma, ürkme gibi davranışlar için bu çiftliklere başvurabilirsiniz.

    Reply
    • By hard March 28, 2021 - 3:00 pm

      We have a team dedicated to Website Design & Development. Our team comprises of a group of experts specializing in Graphic Design, Web-Mobile App Development, User Journey and Digital Marketers. Our talented team designs and develops fully customized websites which are data-driven, user-friendly, speed optimized and built to accelerate growth.
      https://vallettasoft.com/website-design-development-company/

  • By İstanbul Köpek Eğitimi March 25, 2021 - 3:25 pm

    Köpek sahiplenme sorumluluğuna sahip olup da eğitim vermekte zorlanan kişiler için gerekli çözümü köpek eğitim merkezleri sunmaktadır. İstanbul Köpek Eğitim Merkezleri bu konuda yardımcı olmak için çok iyi hizmetler vermektedir.

    Reply
  • By köpek eğitimi March 25, 2021 - 5:58 am

    İnsanların en çok arkadaşlık yaptığı ve aynı ortamı paylaştıkları hayvanlardan birisi köpeklerdir. Dost canlısı olan, sadık bir yapısı bulunan ve oldukça sevecen olan köpekler ile birçok açıdan mutlu bir hayat sürmek mümkündür. Ancak köpeklerin eğitimi konusunda yaşanan çeşitli süreçlerin zorlu olması ya da insanların bu sürelere sahip olamaması, köpek eğitimi İstanbul konusunda yardımcı olan kuruluşları sağlamıştır.

    Reply
  • By Askcorran March 3, 2021 - 1:00 am

    So you are a mangastream fan as you are here searching for alternatives to this website. Mangastream was one of the most favorite websites of all Manga comic readers around the world. But now it is no longer on the internet due to some reasons. Do you know that in today’s time where everything is digital and people love watching series and watch movies on entertainment apps? Yet there are many people like you and me who love reading comics especially manga. Yes, in this age where motion pictures are number one in the list of entertainment, there are comics that are very popular and have a huge fanbase, and one of them are manga comics
    Is MangaStream Dead? Best 26 Alternatives To Read Manga Instantly

    Reply
  • By Askcorran March 2, 2021 - 12:18 am

    We always mistake to understand what is ant radio service? It is always confused that the ANT radio service is the same as Bluetooth in apps and phones. And it is logically correct for us to think so. We think it is just like Bluetooth stack that allows us to connect other phones or devices via Bluetooth connectivity buy it is not so. The ANT radio service is pre-installed in our phones by the manufacture. There may be a chance that if you delete the Bluetooth support on your phone then you might also delete the ANT+ devices. Let us understand what is ant radio service and do you need it?
    what is ant radio service? Do you need it?

    Reply
  • By Jobengg March 2, 2021 - 12:16 am

    2018, 2019 Batch Pass Out freshers has a piece of good news. Accenture is inviting applications from B.E/B.Tech, MCA Freshers for Accenture Off Campus Drive 2020. The drive will be conducted across various locations in India. So, candidates from all the major cities like Hyderabad, Bangalore, NCR, Vijayawada, Pune, Mumbai, Kolkata can take the test for Accenture Off Campus 2020 Drive. The details of the upcoming Accenture Pool Campus Drive are given here. So, all the candidates can check their eligibility, requirements from this page.
    Accenture Off Campus Drive 2020 for 2019, 2018 Batch

    Reply
  • By Techyxl March 2, 2021 - 12:13 am

    QuickBooks is a successfully famous program that enables small and medium businesses to run their accounting processes smoothly. The software comes in different variations which include Premier, Pro, and Enterprise. All of these variants are excellent and can assist streamline a company’s financial and accounting operations.
    The reason why you should use QuickBooks Premier Hosting in 2021

    Reply
  • By Anamounto March 2, 2021 - 12:08 am

    Are you searching for How to disable avast? But you are not able to find a correct process for it. Do not take any tension coz here I have mentioned the correct way to disable the Avast. You will see the different methods on how to close the avast antivirus on your computer, mobile, and Laptops. You will also find the process for how to temporarily disable avast? So read this informative article further for more information on how to disable avast properly.
    How To Disable Avast? Windows, Mac, And Android

    Reply
  • By FlyingRider March 2, 2021 - 12:03 am

    Here is a list of the best smart TVs under 20000 with the description and specification, its pros and cons, the verdict, and various other aspects. This gives you a good idea about which best smart TV under 20000 will suit your needs and comes within your budget.
    Top 10 Best Smart TVs Under 20000 In India To Buy (2021)

    Reply
  • By web desing February 28, 2021 - 4:24 pm

    On our Hello website, we share hundreds of different content for you, such as great content, information you need to know. We are sure that this information will affect your life positively. You can visit our website to be informed about our great ideas.
    Don’t forget to turn on notifications! https://phosphorousideas.com/

    Reply
  • By jain February 22, 2021 - 4:23 am

    great

    Reply

Leave a reply